Scheduled and event-driven compliance workflows at the API gateway

Enlarge

Most compliance tasks in regulated industries are not one-off events. They are recurring: daily API usage reports for internal risk review, weekly audit log exports to the SIEM, real-time transaction threshold checks, hourly credential expiry scans. In most organizations these tasks are scattered across cron jobs on individual servers, scripts triggered by CI pipelines, and manual processes owned by whoever last touched the system. When an auditor asks for evidence that a specific API gateway scheduled compliance workflow ran at the expected time, the answer is usually a mix of shell history and calendar notes — none of which constitutes an audit trail.

The API gateway is where this work belongs. Every compliance check that involves API traffic (usage volumes, access patterns, authentication failures, payload validation) requires access to data the gateway already holds. Running scheduled compliance workflows inside the gateway means the scheduling, execution, and configuration of each task is subject to the same governance controls as the API proxies themselves: RBAC-gated configuration, an immutable audit log of every change, and the same OIDC identity chain from setup through execution.

Zerq's workflow builder supports three entry trigger types beyond HTTP: a cron trigger for scheduled execution, a Kafka consumer for event-driven processing, and an IMAP trigger for email-driven workflows. This post walks through how platform engineers in banking, fintech, and healthcare use cron and Kafka triggers to move compliance automation into the API gateway layer, and what that means for audit evidence when regulators ask questions.

Why compliance tasks outside the gateway create audit problems

The standard setup for recurring compliance tasks is a cron job on a server or a scheduled pipeline stage. This has two structural problems.

The task runs outside the governance boundary. A shell script triggered by a system cron daemon operates under the OS user that owns it. There is no RBAC model, no role requirement to change the schedule, and no record of who approved the last configuration change. The audit trail begins when the script runs and ends when it finishes; there is no record of who changed the cron expression from 0 8 * * * to 0 9 * * *, or why, or who authorized it.

The task is not co-located with the data it processes. A compliance report that needs API traffic counts, client quota consumption, and authentication failure rates must connect to the same datastore the gateway uses. A script that does this is a second consumer of the same database with its own credential, its own connection pool management, and its own failure modes. The gateway already has this data and already knows how to access it.

Tools like Kong, Apigee, and AWS API Gateway are pure request/response proxies. They have no native workflow scheduling and no event-driven trigger system. If you want scheduled compliance automation on those platforms, you are building it outside them: back to the cron job problem. Separate infrastructure, separate audit trail, separate failure modes to manage.

How Zerq's scheduled and event-driven workflow triggers work

A workflow in Zerq is a directed graph of connected nodes attached to a proxy. The entry point of the graph is a trigger node. For HTTP API traffic, the trigger is an http_trigger. For scheduled execution, it is a cron_trigger. For event-driven processing, it is a kafka_consumer. The rest of the workflow is the same regardless of trigger type: validate, transform, call backends, query databases, send notifications, and handle errors with explicit branches.

Cron trigger

The cron_trigger starts a workflow on a fixed schedule. Configuration:

{
  "id": "cron_1",
  "type": "cron_trigger",
  "data": {
    "config": {
      "cron_expression": "0 8 * * 1-5",
      "timezone": "Europe/London",
      "enabled": true
    }
  }
}

• cron_expression: 5-field cron in standard format; here, 8:00 AM on Monday through Friday
• timezone: IANA timezone identifier; the schedule evaluates in the configured timezone, not UTC
• enabled: disable the schedule without removing the workflow configuration

In multi-replica deployments, Zerq uses a Redis distributed lock to guarantee at-most-once execution per schedule slot. The lock key format is workflow_cron_lock:{proxyId}:{nodeId}:{slot}. Only the pod that acquires the lock runs the workflow; other pods skip silently. If Redis coordination is unavailable, the scheduler fails closed rather than risking duplicate execution; this matters in regulated environments where a compliance report running twice can cause operational confusion.

The trigger makes these fields available to downstream nodes: triggered_at (RFC3339 timestamp), trigger (always cron), schedule, timezone, and run_id.

Kafka consumer trigger

The kafka_consumer starts a workflow for each message consumed from a Kafka topic:

{
  "id": "kafka_consumer_1",
  "type": "kafka_consumer",
  "data": {
    "config": {
      "credentials_id": "cred_kafka_prod",
      "topic": "payments.transactions",
      "consumer_group": "zerq-compliance-consumer",
      "enabled": true,
      "commit_mode": "after_success",
      "poll_timeout_ms": 1000
    }
  }
}

• credentials_id: reference to a Kafka credential (bootstrap servers, optional SASL/TLS)
• topic: the Kafka topic to consume from
• consumer_group: consumer group ID for offset tracking and partition assignment
• commit_mode: after_success commits the offset only when the workflow completes successfully; always commits regardless of outcome
• poll_timeout_ms: consume window before the poll loop refreshes

The after_success commit mode matters for compliance: if the workflow fails (validation error, database write failure, HTTP error reaching the reporting endpoint), the offset is not committed. Kafka will redeliver the message on the next poll cycle. No transaction silently disappears because of a transient failure.

In multi-pod deployments, Kafka's native consumer group coordination distributes partitions across pods. At-least-once delivery semantics apply, so downstream actions with side effects should be idempotent when duplicates matter.

The trigger makes these fields available downstream: topic, partition, offset, key, message (JSON object if parseable, otherwise raw string), headers, and run_id.

Building a daily API compliance report

Here is a concrete workflow: every weekday at 8:00 AM London time, Zerq calls an internal analytics endpoint to fetch the previous day's API traffic summary, formats it, and emails it to the compliance team.

Step-by-step setup

1. Create a proxy in any collection. This proxy exists only as the workflow host; it needs no HTTP route if you do not want one. Name it daily-compliance-report.

2. Open the workflow builder. Go to Collections → select the collection → click the proxy → click Edit Workflow.

3. Replace the http_trigger. Delete the default http_trigger node. Open the node palette and add a cron_trigger. Set cron_expression to 0 8 * * 1-5 and timezone to Europe/London.

4. Add an HTTP request node to call your internal analytics API and retrieve the previous day's API usage data. Configure the URL using the triggered_at output of the cron trigger to derive the date range.

5. Add a set_node to reshape the analytics response into a plain-text report body. Map fields from the API response to readable labels:

{
  "id": "n_format",
  "type": "set_node",
  "config": {
    "assignments": {
      "report_date": "{{ $json['cron_1'].triggered_at }}",
      "total_requests": "{{ $json['n_analytics'].response.body.total }}",
      "auth_failures": "{{ $json['n_analytics'].response.body.auth_failures }}",
      "report_body": "API usage report for {{ $json['cron_1'].triggered_at }}\nTotal requests: {{ $json['n_analytics'].response.body.total }}\nAuth failures: {{ $json['n_analytics'].response.body.auth_failures }}"
    }
  }
}

6. Add an smtp_node to deliver the report:

{
  "id": "n_smtp_report",
  "type": "smtp_node",
  "config": {
    "to": ["[email protected]", "[email protected]"],
    "subject": "Daily API compliance report",
    "text": "{{ $json['n_format'].report_body }}"
  }
}

7. Connect the nodes: cron_trigger → http_request_node → set_node → smtp_node.

8. Enable the workflow via the Workflow Enabled toggle on the proxy detail page.

Every time you change the cron schedule, update the recipient list, or modify the report structure, the change appears in the Zerq audit log as an UPDATE to the proxy resource, recording the actor ID, timestamp, IP address, and the full workflow definition.

Processing payment events for real-time compliance

The Kafka consumer trigger is the right pattern when compliance checks need to run for each transaction as it occurs, not on a batch schedule. A common use case in fintech: flag transactions above a reporting threshold and forward them to the compliance system before end of day.

Workflow structure:

kafka_consumer → validate_node → condition_node
                                       ↓               ↓
                              large_transaction    standard
                                       ↓               ↓
                          postgres_node + http_req   postgres_node

The validate node checks that each consumed message contains the required fields:

{
  "id": "n_validate",
  "type": "validate_node",
  "inputs": {
    "value": "{{ $json['kafka_consumer_1'].message }}"
  },
  "config": {
    "schema": {
      "type": "object",
      "required": ["transaction_id", "amount", "currency", "counterparty"],
      "properties": {
        "transaction_id": { "type": "string" },
        "amount":         { "type": "number" },
        "currency":       { "type": "string", "pattern": "^[A-Z]{3}$" },
        "counterparty":   { "type": "string" }
      }
    }
  }
}

If validation fails, the invalid branch routes to a response or error handler. No malformed event reaches the compliance database.

The condition node routes on the threshold:

{
  "id": "n_threshold",
  "type": "condition_node",
  "config": {
    "conditions": [
      {
        "condition": "$json['kafka_consumer_1'].message.amount >= 10000",
        "output": "large_transaction"
      }
    ],
    "default_output": "standard"
  }
}

For the large_transaction branch, a postgres_node writes the transaction record to the compliance database and an http_request_node POSTs to the regulatory reporting endpoint. For the standard branch, a separate postgres_node archives the record to the transaction log.

Because commit_mode is after_success, the Kafka offset is only committed after both the database write and the HTTP call to the compliance system succeed. A transient failure on either means the message is redelivered and the workflow retries.

What the audit trail captures for scheduled workflows

Every cron and Kafka workflow in Zerq is a workflow definition attached to a proxy. Changing the schedule expression, modifying the recipient list, adjusting the transaction threshold, or updating the Kafka topic are all updates to the proxy resource. Each update generates an entry in the Zerq audit log:

The request_body field contains the complete workflow graph at the time of the change. You can reconstruct exactly what the compliance workflow looked like on any given date by reviewing the sequence of UPDATE events for that proxy.

Compliance teams with the Auditor role can read all audit log entries without the ability to modify platform configuration — separation of duties is enforced at the access control layer. The Auditor role exists specifically so the people reviewing compliance evidence cannot be the same people who configure the workflows.

What this looks like in practice

A European bank running Zerq for PSD2 and open banking APIs had two recurring compliance requirements: daily API usage summaries for the internal risk committee, and real-time flagging of high-value transactions for the regulatory reporting system.

Before Zerq, both tasks ran outside the API platform. The daily report was a Python script on a VM with a system cron entry. The transaction monitor was a standalone Kafka consumer application deployed separately. Neither produced a proper audit trail. When an auditor asked who changed the transaction threshold from EUR 10,000 to EUR 15,000 in February and on whose authority, the answer required Git archaeology and an email chain.

After migrating both into Zerq workflows, answering the same question is a single audit log query: filter by resource_type = proxy, action = UPDATE, and the proxy ID of the threshold workflow, scoped to the relevant date range. The result shows the actor ID, the IP address, the exact timestamp, and the complete before state of the workflow definition, including the threshold value. No reconstruction required.

The cron workflow gained at-most-once execution guarantees it never had on the VM. The Kafka consumer gained automatic retry semantics from commit_mode: after_success. Both gained the RBAC gates, audit log entries, and observability dashboards of the API gateway platform. The compliance team can query both the request logs and the workflow configuration history through the same interface they use for everything else.

Bringing scheduled automation into the governance boundary

Zerq's cron and Kafka triggers do not add a new tool to your compliance stack. They bring scheduled and event-driven automation into the same governance boundary as your API proxies: same RBAC, same audit log, same observability, same configuration management.

The practical effect for platform engineers in regulated industries: every compliance automation task becomes a first-class API platform configuration. Scheduled reports, event-driven transaction checks, credential expiry alerts, and audit log exports all live alongside your API catalog, all have a complete change history, and all are accessible to compliance teams through the same tools they already use for evidence gathering.

Zerq is an enterprise API gateway built for regulated industries — one platform for API management, AI agent access, compliance audit, and developer portal, running entirely in your own infrastructure. See how it works or request a demo to walk through your specific requirements.