Skip to main content

AI gateway, LLM gateway or agent API gateway: which one do you actually need?

An AI gateway can mean an LLM gateway for outbound model calls or an agent gateway for inbound API calls. How they differ, where they meet, and which you need.

  • ai
  • ai-gateway
  • mcp
  • security
Zerq team

Two vendor decks land on a CISO's desk in the same week, and both are titled "AI gateway". The first promises token budgets, model fallback, prompt logging and content guardrails. The second promises agent identity, per-operation scope, rate limits and a log of every call an agent makes into your systems. Procurement wants to know which one to buy. The risk committee wants one line in the register that says AI traffic is governed.

Both decks are honest. They describe different products that share a name. An LLM gateway sits in front of model providers and governs the calls your applications make out to them. An agent-facing gateway sits in front of your business APIs and governs the calls AI agents make in. Buy one, tick the box, and the other direction stays open. This post separates the two, shows where they meet, and says plainly which side Zerq covers.

Why the AI gateway label stopped meaning one thing

The label drifted because vendors converged. Azure API Management describes its AI gateway as capabilities on its existing API gateway, not a separate offering: token limits, semantic caching and prompt moderation on the model side, plus the ability to expose REST APIs as MCP servers. Kong splits the territory across plugins: AI Proxy Advanced for routing and fallback across providers, AI Rate Limiting Advanced for token-based limits, and the AI MCP Proxy, which lets agents call existing APIs over MCP. AWS AgentCore Gateway puts MCP, HTTP and inference targets behind one service, and LiteLLM pairs an OpenAI-compatible model proxy with an MCP gateway.

None of this is wrong, but a feature matrix titled "AI gateway" can be deep in one direction and thin in the other. A buyer comparing Kong, Azure API Management, AWS and a dedicated LLM proxy line by line ends up weighing token budgets against tool ACLs as though they answered the same question. They do not: each control has a different owner, produces different evidence and prevents a different failure.

What is the difference between an AI gateway and an LLM gateway?

Strip the branding and look at the direction of the call.

The LLM gateway governs what leaves

An LLM gateway sits between your applications and the model providers. It picks the provider and model, counts tokens against a budget, checks prompts and completions against content rules, may answer from a cache, and logs the exchange. Its unit of control is the model call. The risks it addresses are cost runaway, provider outages, unsafe output and sensitive data leaving for a third-party provider. Its natural owners are the team that pays the inference bill and the data protection officer.

The agent-facing gateway governs what comes in

An agent-facing gateway, often sold as an MCP gateway, sits in front of your business APIs. The caller is an agent: a coding assistant, a support bot, an autonomous workflow. The gateway decides whether this agent identity may call this operation, with this method, at this rate, and records what happened. Its unit of control is the API operation and the identity behind it.

The risks are the ones the OWASP Top 10 for Agentic Applications files under goal hijack and tool misuse: an agent using legitimate tools in ways nobody intended. The mitigation is least privilege, enforced where the call lands. A prompt guardrail can flag an injected instruction, but it cannot tell a legitimate refund call from an abusive one, because both are well-formed requests. Only the gateway in front of the refund API knows whether this identity may POST to it. For a worked payments example, see AI agents in open banking.

The MCP authorization spec points the same way: when authorization is used over HTTP, the MCP server acts as an OAuth 2.1 resource server that must reject tokens not issued for it. Enforcement sits on the server side, which in an enterprise means the gateway in front of your APIs (more in MCP solves connectivity, not governance).

Why a CISO needs to know which AI gateway is which

The risk register has two entries, not one. "Customer data sent to an external model provider" and "agent performs an unauthorized action on a system of record" have different owners, controls and evidence. The first is answered with prompt logs, redaction rules and provider contracts. The second is AI agent API governance: per-identity scopes, deny records (401, 403, 405, 429) and a request log that ties every call to a named identity.

Confusing them fails in predictable ways. A team that bought an LLM gateway reports AI traffic as governed while agents call the payments API through a shared service account. A team that bought an agent gateway reports the same while every prompt, including customer records an agent just fetched, goes to a cloud model unredacted. Both reports are true about one direction and silent about the other.

Where the two gateways meet: one agent loop, two legs

Put both in one diagram and the boundary is obvious. A typical agent turn, plus the control plane, looks like this:

 OUTBOUND (model traffic)
 Agent runtime --(1) prompt + tool list--> LLM gateway --> Model provider
 Agent runtime <--(2) completion: "call execute_endpoint"-- LLM gateway

 INBOUND (agent traffic)
 Agent runtime --(3) MCP tools/call--> Zerq Gateway MCP (/mcp)
     --> /mcp checks: X-Client-ID, X-Profile-ID, bearer token,
         IP allowlist, allowed methods
     --> execute_endpoint via the gateway path: IP allowlist,
         collection access, allowed methods, client policy
     --> upstream business API, request log entry written
 Agent runtime <--(4) tool result-- Zerq Gateway MCP

 NEXT TURN: the tool result from (4) is part of the prompt at (1)

 CONTROL PLANE (inbound changes to the gateway itself)
 Operator agent or Zerq Copilot --(5) OIDC bearer--> Management MCP (/api/v1/mcp)
     --> role check (viewer, modifier, auditor, admin) --> audit log entry

Hops 1 and 2 belong to the LLM gateway, hops 3 and 4 to the agent-facing gateway, and hop 5 is a second inbound path that changes the gateway's own configuration.

The two legs share data. Whatever the agent gets back at hop 4 becomes part of the prompt at hop 1 on the next turn. So the agent gateway's scope decision is also a data-egress decision: a client assigned only the collections a task needs limits what can ever reach the model provider, and the LLM gateway's redaction rules become the second line of defence, not the only one.

They should share identity too: one Zerq client and profile per agent, and one LLM gateway key named after it. When an incident starts with "the agent did something odd", you can line up the model exchange and the API call by agent and time.

How Zerq governs the agent-facing side

Zerq is an agent-facing gateway. Gateway MCP runs at /mcp and exposes exactly four tools: list_collections, list_endpoints, endpoint_details and execute_endpoint. An agent authenticates with the same headers as any REST client (X-Client-ID, X-Profile-ID and Authorization: Bearer), so it only discovers the published collections assigned to its client. Gateway MCP does not implement the spec's OAuth discovery flow. It authenticates agents with the profile's auth method (token, jwt, oidc or mtls) and the client and profile headers. OAuth protected-resource metadata is served only for Management MCP. The profile's IP allowlist and allowed methods are checked when the agent's request reaches /mcp, and again when execute_endpoint calls back through the gateway, where Zerq sees the backend's own address. Leave IP Address Restrictions empty on agent profiles (or allowlist that backend address) and enforce agent egress ranges at your ingress. On that gateway path, collection access and the client's policy (a rate limit plus a quota) also apply, and every call is written to the request log. The AI agents and MCP access use case walks through the setup.

A real Gateway MCP tools/call exchange

Here an agent tries to issue a refund. Its client is assigned only the read-only payments-reporting collection, not the refunds collection. Its profile allows GET and POST: every MCP message is itself a POST to /mcp, so a Gateway MCP profile must allow POST. For an MCP agent, read-only scope comes from which collections its client is assigned, not from allowed methods. The session was opened with initialize, which returned the Mcp-Session-Id header.

curl -X POST "https://gateway.example.com/mcp" \
  -H "Content-Type: application/json" \
  -H "X-Client-ID: 665f1c2ab4e9d03a7c1e5f20" \
  -H "X-Profile-ID: 665f1c4eb4e9d03a7c1e5f21" \
  -H "Authorization: Bearer $AGENT_TOKEN" \
  -H "Mcp-Session-Id: $MCP_SESSION_ID" \
  -d '{
    "jsonrpc": "2.0",
    "id": 7,
    "method": "tools/call",
    "params": {
      "name": "execute_endpoint",
      "arguments": {
        "method": "POST",
        "path": "/refunds/v1/refunds",
        "body": "{\"charge_id\":\"ch_1042\",\"amount\":4999}"
      }
    }
  }'

{
  "jsonrpc": "2.0",
  "id": 7,
  "result": {
    "content": [
      {
        "type": "text",
        "text": "{\"body\":\"{\\\"error\\\":\\\"No API route found for this path\\\"}\",\"headers\":{\"Content-Type\":[\"application/json\"]},\"status_code\":404}"
      }
    ],
    "isError": true,
    "cause": { "message": "Gateway returned 404" }
  }
}

The text field carries the gateway's status_code, body and headers (trimmed here). The denial came from the gateway, not the model: the agent's client is not assigned the refunds collection, so for this client the route does not exist and the POST was refused with 404 before it reached the refunds backend. isError: true means the agent sees a failed tool call, not a silent success. Zerq tags the forwarded request with X-Gateway-Source: mcp, and the request log records it against the agent's Client ID and Profile ID.

Management MCP: the other inbound path

Management MCP is for agents that change the gateway itself: collections, proxies, workflows, policies, clients and profiles. It requires an OIDC bearer token and applies the console's roles: viewer reads, modifier creates, updates and deletes, and only admin can delete log records. Changes land in Audit Logs, reviewed through the read-only auditor role. There is no built-in maker-checker approval for these writes; separation of duties comes from the roles, and approval gates belong in your change process. The security model covers the role split. For role mapping, draft staging and audit rules for agent writes, see AI agent change control.

Zerq Copilot: the one place Zerq calls a model

Zerq Copilot is the only Zerq component that calls an LLM, and it uses the MCP surfaces above as its tools. Model configuration is server-side: AI_MODELS lives in the deployment environment, only id, label and provider reach the browser, and calls run in the Next.js /api/ai/chat route. Providers include OpenAI, Anthropic, Azure OpenAI, Amazon Bedrock, Ollama and OpenAI-compatible endpoints through provider custom with a baseUrl, so an OpenAI-compatible LLM gateway you already run can sit in that slot.

What Zerq does not do

Zerq is not a model-routing proxy for application traffic. It does not route your applications' prompts across providers, enforce token budgets (its policies count requests, not tokens), log prompts and completions for your apps, serve semantic cache hits, or run content guardrails on model input and output. If you need those controls, you still need an LLM gateway, and Zerq sits beside it on the other leg of the loop. Note too that execute_endpoint calls land in Request Logs while Audit Logs hold configuration changes, so plan evidence exports for both.

A checklist for deciding which AI gateway you need

You need an LLM gateway if:

  • Your applications call more than one model provider, or need failover between them.
  • Finance wants token budgets and spend reporting per team or application.
  • Prompts may carry regulated data that must be redacted or logged before it leaves.
  • You need content moderation on prompts or completions.

You need an agent-facing gateway if:

  • Agents call your internal or partner APIs, through MCP or directly.
  • Each agent needs its own identity, scoped to specific collections and HTTP methods.
  • You must show who called which operation, when, and whether it was denied.
  • Agents change gateway configuration, and those changes need role checks and an audit trail.

You need both if your agents call models and your APIs in the same loop. Then ask any vendor selling an AI gateway for enterprises:

  • Which direction does each feature govern: calls out to model providers, or calls into our APIs?
  • Are rate limits counted in tokens, requests or both?
  • Where is an agent's API call authorized, and what does a denied call return?
  • Does each agent get its own identity, and can logs be filtered by it?

What this looks like in practice

Consider an insurer rolling out an internal claims assistant. Security approves it once an LLM gateway sits in front of the cloud model, with prompt logging, a token budget and a PII filter. The assistant fetches claim records through a service account with read and write access to the whole claims API, because that is the quickest way to ship. The register says AI traffic is governed, yet nothing checks what the assistant does with the claims API, and the only record is the backend's access log under the service account's name.

After a review, the claim read endpoints are published as their own Zerq collection, separate from the update endpoints. The assistant gets its own client, assigned only to the read collection, with a policy setting a per-minute rate limit and a daily quota. It connects through Gateway MCP, and a test call to an update endpoint returns 404, logged against its Client ID and Profile ID. The LLM gateway stays, its key renamed to match the Zerq client. And because the assistant reaches only the endpoints in its collection, less customer data reaches the model at all.

Two jobs, one name

An LLM gateway governs outbound calls to model providers: routing, token budgets, prompt logging, caching and guardrails. An agent-facing gateway governs inbound calls into your business APIs: identity, scope, rate limits and logs. Zerq does the second through Gateway MCP and Management MCP and does not do the first. Decide by direction, map agent identities across both, and treat the agent gateway's scope as your first control on what data reaches a model.

Zerq is an enterprise API gateway built for regulated industries — one platform for API management, AI agent access, compliance audit, and developer portal, running entirely in your own infrastructure. See how it works or request a demo to walk through your specific requirements.

Request Enterprise DemoExplore capabilities