An AI gateway is a proxy that sits between the people, applications, and agents that call AI models and the providers that serve them, so every request passes through one place on its way to the model and every response passes back through it. At minimum, that place handles routing across providers and models, authentication, rate limits, cost tracking, logging, caching, and fallbacks. Beyond the minimum, the term splits into two products that share a name and not much else, which is why the search results for it read like an argument.
The short version: an AI gateway is the one point every AI request passes through. Some teams buy it to route traffic and manage spend. Others buy it to decide, per person and per agent, what is allowed and to keep the record. Know which one you are shopping for.
What an AI gateway does
The layout is simple: callers on one side, models on the other. The caller might be a customer-facing application, an internal tool, an employee in a chat interface, or an autonomous agent working through a task list. The model might be hosted by OpenAI, Anthropic, or Google, or be an open-weight model on your own hardware. The AI gateway presents one endpoint to the callers, holds the provider credentials itself, and forwards each request to whichever model the rules say it should go to.
The base feature set is consistent across vendors:
- Routing and fallbacks: send a request to a model by name or by rule, and retry against a second provider when the first one times out or rate-limits you.
- Authentication and rate limits: callers present a key or token to the gateway rather than to the provider, and the gateway decides how much traffic each key may generate.
- Cost tracking: token counts by caller, team, model, and application, turned into spend reports and budget caps.
- Logging: the prompt, the response, latency, token usage, and the routing decision, kept for debugging and review.
- Caching: identical or semantically similar requests answered from cache instead of billed again.
API gateway vs AI gateway
Same pattern, different payload. An API gateway terminates TLS, authenticates the caller, applies rate limits, and forwards a request to a backend service. It treats the request body as opaque, and knowing the path, method, headers, and size is enough for its job.
An AI gateway keeps all of that and adds an understanding of the payload: the prompt, the system message, the target model, the tools the model may call, and the expected token count. That knowledge lets it apply rules to the content and cost of a request as well as its shape. A rule like "no more than 500 requests per minute" belongs to both kinds of gateway. A rule like "this caller may not send claims data to a model outside our tenant," or "this agent may not invoke the payments tool," only makes sense at an AI gateway. Many teams run one behind the other, with the API gateway handling ingress for everything and the AI gateway handling the traffic bound for a model.
Two kinds of AI gateway, and the term is overloaded
Two categories of product carry the name, they are built for different buyers, and a team that buys one expecting the other usually finds out during its first audit.
The developer routing layer. Built for engineering teams shipping AI features. Its concerns are model routing, load balancing across providers, semantic caching, spend guardrails, retries, and observability for the requests an application generates. The unit of policy is an API key: this key may use these models, up to this budget, at this rate. Kong AI Gateway, Cloudflare AI Gateway, Portkey, LiteLLM, TrueFoundry, and Databricks Mosaic AI Gateway are examples of this category. Several vendors label this layer a "control plane" for LLM traffic. It is the right tool when the question is how to get a model into production reliably and without surprise invoices.
The governance point. Built for the organization that has to answer a different question: who used what AI, on what data, and was it allowed. This kind of AI gateway is identity-aware, so it knows which person or which agent is behind a request, usually by integrating with the identity provider. It enforces policy per request: which models this role may reach, which classes of data may leave the tenant, which tools an agent may call, which actions need a human sign-off before they run. It writes an audit trail entry for every request, and it has a kill switch that stops a person, an agent, or an application from making further calls without redeploying anything. The unit of policy is a human or agent identity rather than a key.
| Routing layer | Governance point | |
|---|---|---|
| Primary user | Platform and application engineers | Security, compliance, and risk leads, alongside IT |
| Unit of policy | API key or application | Human or agent identity, mapped from the identity provider |
| What it inspects | Model name, token counts, latency, cache hits | Who is asking, what data is in the request, which tools and actions are requested |
| What it produces | Routing decisions, spend reports, traces | Per-request allow or deny decisions, an audit record, alerts, a kill switch |
| What it is for | Shipping AI features reliably and within budget | Proving AI use stayed inside policy, per person and per agent |
Some products do both in part. A routing layer often has a logging feature, and a governance point often routes across providers. The overlap is where buyers get confused. Ask which job you are hiring the AI gateway for, then check whether the product's unit of policy matches that job. A gateway that only knows API keys cannot tell an auditor which employee sent a claims file to a model, no matter how good its dashboards are.
Do I need an AI gateway?
Three situations where the answer is yes.
You call more than one model or provider. Once two applications talk to two providers, credentials, retries, and spend tracking are already duplicated. The routing layer earns its place here even with no governance requirement at all.
Employees or agents touch regulated data through AI. Claims files, patient records, account numbers, anything under a data-handling rule. The moment that data can reach a model, someone has to be able to say which model, sent by whom, under which policy. Per-key logging cannot answer that, and a governance point can, while also giving people an approved path that is easier than a personal account. That path does more to reduce shadow AI than device monitoring does.
You owe evidence of AI use to a regulator, an auditor, a board, or a customer. Carriers under the NAIC model bulletin, health systems, lenders, and any vendor filling in a security questionnaire are already in this position. An acceptable use policy states the rules, and the AI gateway is where the rules are applied and where the record comes from.
One situation where it is premature: a single team, a single provider, prototype-stage, no regulated data in the prompts. Add the gateway when the second provider or the first sensitive dataset shows up. In our experience that is sooner than the team expects.
What are the most popular AI gateways?
For the routing layer, the products that come up most often are Kong AI Gateway, Cloudflare AI Gateway, Portkey, LiteLLM, TrueFoundry, and Databricks Mosaic AI Gateway. They differ on deployment model (hosted service, self-hosted proxy, or a library inside your application), on provider coverage, and on depth of observability. We are not ranking them and have no market-share figures to offer.
The governance-point category is newer and smaller. Swept is in it, and the platforms that describe themselves as AI posture management or runtime AI governance tend to land here too. The test for membership is the one in the table: does the product know who is asking, and can it say no per request.
AI gateway architecture in one paragraph
The gateway sits inline between callers and providers, so the application's model endpoint points at the gateway and the gateway holds the provider keys. To do the governance job it needs four connections: the identity provider (so a request maps to a person, role, or agent), a policy store (the rules it enforces), the model providers (where it forwards approved traffic), and a log sink (where per-request decisions and evidence go). Deployment is either a network proxy that every request passes through, or a sidecar or SDK embedded next to the application. The proxy form is easier to make universal, and the SDK form is easier to adopt one app at a time. Inline evaluation adds latency on the order of the identity lookup and the policy check, typically tens of milliseconds, which is small against a model call that takes seconds. As with any proxy, an unavailable gateway blocks every AI request behind it, so availability deserves the same attention it gets for an API gateway.
The MCP gateway
Agents that call tools through the Model Context Protocol raise the same question one level up. Where a chat request asks which model a person may use with what data, a tool call asks which tool an agent may invoke, with which arguments, on whose behalf. An MCP gateway is the same governance point applied to tool traffic: agent identity, per-call policy, an audit record for every invocation, and a way to stop an agent mid-task. Teams that already run an AI gateway for model calls usually extend it to tool calls rather than standing up a separate product, since the policy and the identity mapping are the same. The broader discipline is covered in AI agent governance.
Why the governance reading matters
A routing layer is a platform team's purchase. The governance point is what compliance and security leads have started asking about, and that is why the term has spread beyond engineering over the past two years. The policy check, the audit record, and the kill switch all live at the same point in the path, the one every request already passes through. "Govern AI at the point of use" is a statement about architecture: enforcement happens where the request is, which is what runtime security and AI guardrails both need in order to be real. That is also what separates an AI governance program that produces evidence from one that produces documents.
Swept is an AI gateway of the second kind. It connects to your identity provider, applies the limits you set to each request from each person and agent, meters spend, and keeps the record across the models and tools your organization has approved. If you are working out where a governed path for employee AI fits, Swept Access is the place to start, and we are glad to talk it through.
What Is FAQs
An AI gateway is a proxy that sits between the applications, employees, and agents that call AI models and the providers that serve them. Every request passes through it. At minimum it handles routing across models and providers, authentication, rate limits, cost tracking, logging, caching, and fallbacks. Some AI gateways go further and enforce per-person or per-agent policy on each request and keep an audit record of the decision.
An API gateway treats the request body as opaque and applies rules to the shape of a request, meaning its path, method, headers, and rate. An AI gateway understands the payload, meaning the prompt, the target model, the token count, and the tools the model may call. That lets it apply rules to the content and cost of a request, such as which data may reach which model, which an API gateway cannot see.
Yes in three situations. You call more than one model or provider. Employees or agents send regulated data through AI. Or you owe evidence of AI use to a regulator, auditor, board, or customer. It is premature for a single team on a single provider with no sensitive data in the prompts, though that stage tends to end sooner than teams expect.
The terms are mostly synonyms. LLM gateway usually refers to the developer routing layer, covering model routing, fallbacks, caching, and spend tracking for applications that call large language models. AI gateway is the broader label and now also covers the governance point, the identity-aware layer that decides per request what a person or agent may do and records the outcome. Look at what the product enforces rather than which label it uses.
An MCP gateway is an AI gateway for tool calls. Agents that use the Model Context Protocol to reach tools raise the same question as model calls, one level up. Which agent may call which tool, with which arguments, on whose behalf? The MCP gateway sits inline on that traffic, applies policy per call, records each invocation, and can stop an agent mid-task.
Only the governance kind can. Enforcing a usage policy requires knowing who is asking, which means integrating with the identity provider, and then applying rules per request about models, data classes, tools, and actions. A routing layer that only knows API keys can cap spend and log traffic, but it cannot tell you whether a specific employee stayed inside the acceptable use policy.