API gateway high availability: what your replicas actually have to agree on

One replica of an API gateway is simple. Every rate limit counter, every idempotency lock, every scheduled workflow lives in one process, and that process is trivially consistent with itself. The moment you run two replicas behind a load balancer, everything the gateway remembers becomes a distributed systems question, and API gateway high availability turns out to be less about adding pods and more about deciding what those pods must agree on.

Platform teams usually discover this the uncomfortable way. A partner reports being rate limited at half their contracted quota, because two replicas each enforced the full limit independently and the load balancer split the traffic. A scheduled reconciliation workflow runs twice at 02:00 because both replicas believed they owned it. An operator disables a workflow in the management UI and it keeps firing, because the replica that held the trigger in memory never heard about the change.

None of these are exotic failures. They are the default behavior of any gateway that scales horizontally without a deliberate coordination design. This post walks through what has to be coordinated, why bolted-on approaches fall short, and exactly how Zerq handles each layer, down to the Redis key names.

Why API gateway high availability is a coordination problem

Legacy gateway architectures answered this question with heavy machinery. Apigee coordinates message processors through a Cassandra ring. Kong's traditional mode clusters nodes around a shared Postgres database, with rate limit counters that are either node-local (fast, wrong under load balancing) or synchronized with a tunable delay (closer, still approximate). MuleSoft pushes the problem into its control plane. The cloud gateways from AWS and Azure solve coordination well, but only inside infrastructure you cannot inspect or self-host, which rules them out for the air-gapped and sovereign deployments that regulated teams actually run.

The failure pattern across all of them is the same: state that should be shared silently defaults to per-node. In-memory rate limit counters multiply your real limit by the replica count. In-memory schedulers duplicate cron executions. In-memory route caches serve stale config after a change. Each one works perfectly in the single-node staging environment and breaks only in production, only under load balancing, which is why these bugs survive review after review.

The honest architectural statement is this: a gateway replica may keep no authoritative state of its own. Every fact that matters, including counters, locks, config, and audit records, must live in a store all replicas share, and the replica processes must be disposable. That is the design Zerq is built around, and the rest of this post shows the concrete mechanics.

The stateless split

Zerq's architecture divides cleanly into disposable and durable components. The gateway backend (a single Go binary), the management UI, and the developer portal are stateless and scale horizontally to any replica count. MongoDB holds all configuration and the full audit trail. Redis holds coordination state: rate limit counters, quota usage, idempotency locks, scheduler locks, and the config change broadcast channel. Your identity provider completes the stateful set.

Everything runs inside your own environment, whether that is on-premises Kubernetes, your cloud account, or a fully offline enclave. There is no external control plane to phone home to, which also means there is no external control plane whose outage becomes your outage. The coordination story below involves exactly two dependencies you already operate: MongoDB and Redis.

The switch from single-replica to multi-replica behavior is one configuration decision. With CACHE_TYPE=memory, each replica keeps its own counters, which is acceptable only when one replica exists. With Redis, all coordination state is shared:

# Coordination store selection.
# "memory" is per-replica and safe only for single-instance deployments.
# "redis" shares all counters, locks, and sync events across replicas.
CACHE_TYPE=redis

# Connection for the shared coordination store.
REDIS_URL=redis.zerq-data.svc.cluster.local:6379
REDIS_USERNAME=zerq-gateway
REDIS_PASSWORD=<from-your-secrets-manager>

# Per-scheduler switches. Leave enabled on all replicas;
# coordination below decides which replica does the work.
WORKFLOW_SCHEDULER_ENABLED=true
WORKFLOW_KAFKA_CONSUMER_ENABLED=true
WORKFLOW_IMAP_TRIGGER_ENABLED=true

The gateway warns loudly if you run coordination-sensitive features on memory cache. The idempotency locker, for example, logs Using memory cache for idempotency locking - NOT SAFE for critical endpoints in multi-pod environments! rather than letting duplicate payment requests slip through quietly.

Rate limits and quotas that mean what they say

When a request arrives, the gateway resolves the client's policy and enforces it with a counter stored in Redis under a sanitized key: rate_limit:client:{clientID} for identified clients, rate_limit:ip:{clientIP} as the fallback. Because the counter lives in the shared store rather than in process memory, a partner with a 100-requests-per-minute policy gets exactly 100 requests per minute whether you run two replicas or twenty, and whichever replica serves the request.

Longer-horizon quotas work the same way with one refinement: usage accumulates in Redis on the hot path, and a background worker flushes usage snapshots to MongoDB every five minutes. Enforcement stays fast, and the durable usage record that billing and observability read survives any replica restart. When either check fails, the client receives a structured 429 with rate_limit_exceeded or quota_limit_exceeded in the body, and the rejection is written to the same audit trail as every other request.

Idempotency protection follows the identical pattern. Duplicate-sensitive endpoints acquire a distributed Redis lock keyed on the idempotency token before executing, so two replicas receiving the same retried payment request cannot both forward it to your backend.

Config changes propagate by broadcast, not by polling

The second coordination problem is configuration drift. A replica that caches route and workflow state in memory must learn about changes immediately, or an operator's change applies only to whichever replica handled their session.

Zerq handles this with a Redis Pub/Sub channel named workflow.trigger.sync. When anyone saves or toggles a workflow, whether through the management UI, the API, or Management MCP, the replica that processes the write publishes an event that every replica, including the publisher itself, consumes:

{
  "event_type": "proxy_changed",
  "proxy_id": "69c6ecc97d9e178b583b9dce",
  "version": 1774658272401990000,
  "source": "gateway-7d9f6c-x2kkr:1:d66fdcf3-cd94-44e2-8f9b-8eec619334b4"
}

The version field is a nanosecond timestamp used to discard stale or duplicate events, and source identifies the originating replica as {hostname}:{pid}:{uuid}. On receipt, each replica re-reads the proxy from MongoDB rather than trusting the event payload, evaluates whether its workflow should be active, and reconciles its local schedulers: start what appeared, stop what was removed, restart what changed. The event is a doorbell; MongoDB stays the source of truth.

Two guards make this safe under real operations. On startup, every replica runs a full bootstrap sweep of MongoDB before relying on the channel, so a freshly rolled pod cannot miss events published before it subscribed. And if Redis drops and reconnects, each replica repeats that full sweep automatically, recovering anything published during the gap. Drift is bounded to the length of a Redis outage, then repaired without operator action.

Scheduled work runs exactly once

Workflows in Zerq can be triggered by cron schedules, Kafka topics, or IMAP mailboxes, and every replica runs every scheduler. That is deliberate: it means any replica can take over any job. What prevents duplicate execution is a coordination mechanism matched to each trigger type.

Cron uses a per-slot distributed lock. When a schedule fires, each replica computes the same key, workflow_cron_lock:{proxyID}:{nodeID}:{slotTime}, with the slot timestamp in RFC 3339 form baked into the key. Exactly one replica acquires it and executes; the others log a debug line and move on. The winner extends the lock in a background loop while the workflow runs, then releases it using a fresh context so the release succeeds even during shutdown.

Kafka needs no gateway-side locking at all, because consumer group partition assignment already guarantees each partition one owner. IMAP triggers combine an ownership lock (imap_owner_lock:{proxyID}:{nodeID}) that elects one replica to poll the mailbox with atomic per-message UID claims (imap_uid_seen), so even an ownership handover mid-poll cannot process the same email twice. A lock busy line in a non-owning replica's logs is not an error; it is the election working.

Probes, rollouts, and the shutdown sequence

High availability is exercised most often not by failures but by your own deployments, and this is where honest health semantics matter. Zerq exposes two distinct probes. /livez answers only "is this process alive" and always returns success while the process runs, so orchestrators never restart a pod for a dependency's problem. /readyz answers "can this replica serve correctly right now" by pinging MongoDB, and Redis when configured, each with a two-second timeout. A replica that cannot reach its coordination store reports unready and receives no traffic, rather than serving requests with unenforceable limits. This closes the classic failure mode where readiness passes too early and traffic routes to an instance that is up but not yet correct.

On the way down, a replica receiving SIGTERM runs an ordered sequence: cancel the scheduler context so every trigger worker stops its loop and releases its locks, close outbound HTTP connections, close the cache connection, disconnect from MongoDB, then drain in-flight requests with a 30-second shutdown timeout. Because lock release happens eagerly rather than waiting for TTL expiry, a surviving replica picks up scheduled work within one poll interval instead of waiting out a stale lock.

Those two behaviors compose into zero-downtime rolling updates. The operational procedure, from the upgrade runbook:

Freeze non-essential config changes and publish anything pending, so the rollout is not racing operator edits.
Record current image tags and deployment revision, and confirm recent backups of MongoDB and your identity provider. This is your rollback boundary.
Define rollback triggers before you start: a 5xx threshold on core partner routes, OIDC login failure for operators or consumers, or a rollout degraded beyond your timeout.
Start the rollout and watch it converge: kubectl -n zerq rollout status deploy/backend. Kubernetes replaces replicas one at a time, gating each on /readyz.
Run acceptance checks: the health endpoint, one protected route with a production client and profile, a management UI login, and a developer portal login.
Watch the dashboards for sustained growth in 401, 403, 429, or 5xx counts through your stabilization window before declaring the upgrade done.

At no point in that sequence does a partner request fail because of the deployment, and every config change made around it lands in the audit trail with the operator's identity attached, which your security reviewers will ask about eventually.

A checklist before you raise the replica count

Before scaling any gateway, including Zerq, past one replica, confirm each of these:

Is CACHE_TYPE=redis set on every replica, with all replicas pointing at the same Redis? Split coordination stores are worse than none.
Is Redis itself deployed for availability, with persistence or a replica, since it now holds your limit counters and locks?
Do rate limits enforce the contracted number under load balancing? Send a burst through the load balancer, not at a single pod, and count the 429s.
Does a config change made in the management UI take effect on every replica within seconds? Toggle a workflow and watch each replica's sync log line.
Do scheduled workflows execute exactly once per slot with all replicas healthy, and still execute when you kill the owning replica mid-window?
Do readiness probes fail when MongoDB or Redis is unreachable, and does your load balancer actually honor that signal?
Does a rolling restart under sustained traffic produce zero client-visible errors?

If you cannot answer yes to all seven, you have a load-balanced set of independent gateways, not a highly available one.

What this looks like in practice

A payments platform we worked with ran their previous gateway as a single production node, not by choice but because their partner rate limits and duplicate-payment protection only worked in one process. Every upgrade was a maintenance window at 03:00, announced to partners a week ahead. Their one attempt at running two nodes ended after a settlement workflow executed twice in a night.

On Zerq they run three gateway replicas on Kubernetes against their existing MongoDB and Redis. Partner limits are enforced from shared counters, so contractual numbers hold regardless of which replica a request lands on. The nightly settlement workflow fires on a cron trigger, and the per-slot lock guarantees one execution even though all three replicas are eligible to run it. Upgrades happen during business hours as ordinary rolling updates, gated on readiness, with the runbook's rollback thresholds agreed in advance. The maintenance window emails stopped, and their audit trail shows an unbroken request record straight through every deployment since.

Closing

High availability for an API gateway is a coordination problem before it is a replica-count problem. Zerq gives you a strict stateless split, so replicas are disposable by design. Shared Redis state, so rate limits, quotas, and idempotency mean the same thing on every replica. Broadcast config sync with bootstrap recovery, so no replica serves stale policy. Per-trigger leader election, so scheduled work runs exactly once. Honest liveness and readiness semantics, plus an ordered graceful shutdown, so rolling updates are invisible to your partners. All of it runs inside your own perimeter, on infrastructure you already operate.

Zerq is an enterprise API gateway built for regulated industries — one platform for API management, AI agent access, compliance audit, and developer portal, running entirely in your own infrastructure. See how it works or request a demo to walk through your specific requirements.