An AI gateway is a proxy that sits between the people, applications, and agents that call AI models and the providers that serve them, so every request passes through one place on its way to the model and every response passes back through it. At minimum, that place handles routing across providers and models, authentication, rate limits, cost tracking, logging, caching, and fallbacks. Beyond the minimum, the term splits into two products that share a name and not much else, which is why the search results for it read like an argument.
The short version: an AI gateway is the one point every AI request passes through. Some teams buy it to route traffic and manage spend. Others buy it to decide, per person and per agent, what is allowed and to keep the record. Know which one you are shopping for.
What an AI gateway does
The layout is simple: callers on one side, models on the other. The caller might be a customer-facing application, an internal tool, an employee in a chat interface, or an autonomous agent working through a task list. The model might be hosted by OpenAI, Anthropic, or Google, or be an open-weight model on your own hardware. The AI gateway presents one endpoint to the callers, holds the provider credentials itself, and forwards each request to whichever model the rules say it should go to.
The base feature set is consistent across vendors:
- Routing and fallbacks: send a request to a model by name or by rule, and retry against a second provider when the first one times out or rate-limits you.
- Authentication and rate limits: callers present a key or token to the gateway rather than to the provider, and the gateway decides how much traffic each key may generate.
- Cost tracking: token counts by caller, team, model, and application, turned into spend reports and budget caps.
- Logging: the prompt, the response, latency, token usage, and the routing decision, kept for debugging and review.
- Caching: identical or semantically similar requests answered from cache instead of billed again.
API gateway vs AI gateway
Same pattern, different payload. An API gateway terminates TLS, authenticates the caller, applies rate limits, and forwards a request to a backend service. It treats the request body as opaque, and knowing the path, method, headers, and size is enough for its job.
An AI gateway keeps all of that and adds an understanding of the payload: the prompt, the system message, the target model, the tools the model may call, and the expected token count. That knowledge lets it apply rules to the content and cost of a request as well as its shape. A rule like "no more than 500 requests per minute" belongs to both kinds of gateway. A rule like "this caller may not send claims data to a model outside our tenant," or "this agent may not invoke the payments tool," only makes sense at an AI gateway. Many teams run one behind the other, with the API gateway handling ingress for everything and the AI gateway handling the traffic bound for a model.
Two kinds of AI gateway, and the term is overloaded
Two categories of product carry the name, they are built for different buyers, and a team that buys one expecting the other usually finds out during its first audit.
The developer routing layer. Built for engineering teams shipping AI features. Its concerns are model routing, load balancing across providers, semantic caching, spend guardrails, retries, and observability for the requests an application generates. The unit of policy is an API key: this key may use these models, up to this budget, at this rate. Kong AI Gateway, Cloudflare AI Gateway, Portkey, LiteLLM, TrueFoundry, and Databricks Mosaic AI Gateway are examples of this category. Several vendors label this layer a "control plane" for LLM traffic. It is the right tool when the question is how to get a model into production reliably and without surprise invoices.
The governance point. Built for the organization that has to answer a different question: who used what AI, on what data, and was it allowed. This kind of AI gateway is identity-aware, so it knows which person or which agent is behind a request, usually by integrating with the identity provider. It enforces policy per request: which models this role may reach, which classes of data may leave the tenant, which tools an agent may call, which actions need a human sign-off before they run. It writes an audit trail entry for every request, and it has a kill switch that stops a person, an agent, or an application from making further calls without redeploying anything. The unit of policy is a human or agent identity rather than a key.
| Routing layer | Governance point | |
|---|---|---|
| Primary user | Platform and application engineers | Security, compliance, and risk leads, alongside IT |
| Unit of policy | API key or application | Human or agent identity, mapped from the identity provider |
| What it inspects | Model name, token counts, latency, cache hits | Who is asking, what data is in the request, which tools and actions are requested |
| What it produces | Routing decisions, spend reports, traces | Per-request allow or deny decisions, an audit record, alerts, a kill switch |
| What it is for | Shipping AI features reliably and within budget | Proving AI use stayed inside policy, per person and per agent |
Some products do both in part. A routing layer often has a logging feature, and a governance point often routes across providers. The overlap is where buyers get confused. Ask which job you are hiring the AI gateway for, then check whether the product's unit of policy matches that job. A gateway that only knows API keys cannot tell an auditor which employee sent a claims file to a model, no matter how good its dashboards are.
Multi-model AI and the AI gateway
Multi-model AI means using several models, often from several providers, and sending each task to the one best suited for it. Very few organizations stay on one model for long. A claims team may want Claude for long coverage documents, an underwriting analyst may prefer ChatGPT for spreadsheet work, a developer may use a code model, and a batch job that classifies inbound email can run on a small, cheap model or an open-weight model inside your own tenant.
The economics push in the same direction. Frontier models are priced for hard reasoning, and most enterprise AI traffic is not hard reasoning. Summarizing a first notice of loss, extracting fields from a declarations page, or drafting a routine policyholder reply rarely needs the largest model available. Right-sizing the model to the task is one of the more reliable ways to keep AI spend in check, and it only works if switching models is cheap.
Without a gateway, every model a team adopts brings its own credentials, SDK, logging format, data-handling terms, and bill. Policy gets written once per provider, if at all. The AI gateway turns multi-model AI into one surface: one endpoint for callers, one place where provider keys live, one log format, one set of rules about which data may reach which model, and one spend report across all of them. For the governance kind, it also means a person's permissions follow them across models instead of being re-created inside each vendor's admin console.
What is an LLM router?
An LLM router is the component that decides which model handles a given request. It is a feature of an AI gateway (usually the routing layer), though some teams run a router as a library inside the application. Routing rules come in a few forms:
- Static rules: a named model per application, team, or task type. Simple and predictable.
- Cost and latency rules: send to the cheapest or fastest model that meets a quality bar, and fail over to a second provider on timeouts or rate limits.
- Content-based routing: classify the request (by length, task type, or difficulty) and send easy requests to a small model and hard ones to a frontier model.
- Policy-based routing: route by data class and identity, so a prompt containing policyholder PII goes only to a model running inside your tenant, regardless of what the caller asked for.
The first three are about cost and reliability, and every routing-layer product offers some version of them. The fourth is where routing meets governance: the router needs to know who is asking and what is in the prompt, which is the governance point's job. When evaluating an LLM router, ask what signals it can route on, whether routing decisions are logged per request, and whether a policy rule can override a caller's model choice.
AI gateways and vendor lock-in
Lock-in with AI vendors is quieter than it was with past enterprise software, and it builds faster. It accumulates in three places:
- Workflows: prompts, agents, and integrations written against one provider's SDK and tool format.
- Context: conversation history, uploaded files, memory, and retrieval indexes stored inside one vendor's product.
- Pricing: per-seat contracts and committed spend that make switching expensive even when a better or cheaper model ships.
The goal is that no single model vendor holds your workflows, context, or pricing hostage. An AI gateway helps because callers integrate with the gateway rather than with each provider, so moving a task from one model to another is a routing change instead of a rewrite. Logs, policies, and usage history stay with you. And because you can switch models per task, you can pay for usage on the model each task needs instead of idle seats on a single vendor's top tier.
The general MLOps advice applies here as well: prefer open standards and portable formats, choose tools that can be replaced, and keep workflows loosely coupled to any single implementation. The model landscape will look different in three years. A stack that assumes today's leader stays the leader is a bet, and the gateway is how you avoid having to make it.
Do I need an AI gateway?
Three situations where the answer is yes.
You call more than one model or provider. Once two applications talk to two providers, credentials, retries, and spend tracking are already duplicated. The routing layer earns its place here even with no governance requirement at all.
Employees or agents touch regulated data through AI. Claims files, patient records, account numbers, anything under a data-handling rule. The moment that data can reach a model, someone has to be able to say which model, sent by whom, under which policy. Per-key logging cannot answer that, and a governance point can, while also giving people an approved path that is easier than a personal account. That path does more to reduce shadow AI than device monitoring does.
You owe evidence of AI use to a regulator, an auditor, a board, or a customer. Carriers under the NAIC model bulletin, health systems, lenders, and any vendor filling in a security questionnaire are already in this position. An acceptable use policy states the rules, and the AI gateway is where the rules are applied and where the record comes from.
One situation where it is premature: a single team, a single provider, prototype-stage, no regulated data in the prompts. Add the gateway when the second provider or the first sensitive dataset shows up. In our experience that is sooner than the team expects.
What are the most popular AI gateways?
For the routing layer, the products that come up most often are Kong AI Gateway, Cloudflare AI Gateway, Portkey, LiteLLM, TrueFoundry, and Databricks Mosaic AI Gateway. They differ on deployment model (hosted service, self-hosted proxy, or a library inside your application), on provider coverage, and on depth of observability. We are not ranking them and have no market-share figures to offer.
The governance-point category is newer and smaller. Swept is in it, and the platforms that describe themselves as AI posture management or runtime AI governance tend to land here too. The test for membership is the one in the table: does the product know who is asking, and can it say no per request.
AI gateway architecture in one paragraph
The gateway sits inline between callers and providers, so the application's model endpoint points at the gateway and the gateway holds the provider keys. To do the governance job it needs four connections: the identity provider (so a request maps to a person, role, or agent), a policy store (the rules it enforces), the model providers (where it forwards approved traffic), and a log sink (where per-request decisions and evidence go). Deployment is either a network proxy that every request passes through, or a sidecar or SDK embedded next to the application. The proxy form is easier to make universal, and the SDK form is easier to adopt one app at a time. Inline evaluation adds latency on the order of the identity lookup and the policy check, typically tens of milliseconds, which is small against a model call that takes seconds. As with any proxy, an unavailable gateway blocks every AI request behind it, so availability deserves the same attention it gets for an API gateway.
The MCP gateway
Agents that call tools through the Model Context Protocol raise the same question one level up. Where a chat request asks which model a person may use with what data, a tool call asks which tool an agent may invoke, with which arguments, on whose behalf. An MCP gateway is the same governance point applied to tool traffic: agent identity, per-call policy, an audit record for every invocation, and a way to stop an agent mid-task. Teams that already run an AI gateway for model calls usually extend it to tool calls rather than standing up a separate product, since the policy and the identity mapping are the same. The broader discipline is covered in AI agent governance.
Why the governance reading matters
A routing layer is a platform team's purchase. The governance point is what compliance and security leads have started asking about, and that is why the term has spread beyond engineering over the past two years. The policy check, the audit record, and the kill switch all live at the same point in the path, the one every request already passes through. "Govern AI at the point of use" is a statement about architecture: enforcement happens where the request is, which is what runtime security and AI guardrails both need in order to be real. That is also what separates an AI governance program that produces evidence from one that produces documents.
For years, AI governance meant 50-page PDF policies, quarterly audit meetings, and manual reviews of chat logs. That model breaks once agents arrive. If you deploy 1,000 autonomous agents to handle customer service, you cannot hire 1,000 people to watch them, and manual compliance checks that take hours or days cannot keep pace with agents that act in milliseconds. Governance moves from policy (what should happen) to protocol (what actually happens): instead of a document that says "agents must not share PII," each output is checked for PII before it leaves your environment. That infrastructure has three parts, and the AI gateway is where all three live:
- Real-time interception: a governance layer that sits between the model and the world.
- Context-aware filtering: rules that understand intent, blocking a persuasive attempt to elicit financial data as well as a literal card number.
- Automated audit trails: every decision, check, and rejection logged and signed, so answering an examiner is a query against the record.
Good brakes are what let you drive fast. Teams with enforcement built into the path ship more AI, because they know where the limits are.
Swept is an AI gateway of the second kind. It connects to your identity provider, applies the limits you set to each request from each person and agent, meters spend, and keeps the record across the models and tools your organization has approved. If you are working out where a governed path for employee AI fits, Swept Access is the place to start, and we are glad to talk it through.
What Is FAQs
An AI gateway is a proxy that sits between the applications, employees, and agents that call AI models and the providers that serve them. Every request passes through it. At minimum it handles routing across models and providers, authentication, rate limits, cost tracking, logging, caching, and fallbacks. Some AI gateways go further and enforce per-person or per-agent policy on each request and keep an audit record of the decision.
An API gateway treats the request body as opaque and applies rules to the shape of a request, meaning its path, method, headers, and rate. An AI gateway understands the payload, meaning the prompt, the target model, the token count, and the tools the model may call. That lets it apply rules to the content and cost of a request, such as which data may reach which model, which an API gateway cannot see.
Yes in three situations. You call more than one model or provider. Employees or agents send regulated data through AI. Or you owe evidence of AI use to a regulator, auditor, board, or customer. It is premature for a single team on a single provider with no sensitive data in the prompts, though that stage tends to end sooner than teams expect.
The terms are mostly synonyms. LLM gateway usually refers to the developer routing layer, covering model routing, fallbacks, caching, and spend tracking for applications that call large language models. AI gateway is the broader label and now also covers the governance point, the identity-aware layer that decides per request what a person or agent may do and records the outcome. Look at what the product enforces rather than which label it uses.
An MCP gateway is an AI gateway for tool calls. Agents that use the Model Context Protocol to reach tools raise the same question as model calls, one level up. Which agent may call which tool, with which arguments, on whose behalf? The MCP gateway sits inline on that traffic, applies policy per call, records each invocation, and can stop an agent mid-task.
Only the governance kind can. Enforcing a usage policy requires knowing who is asking, which means integrating with the identity provider, and then applying rules per request about models, data classes, tools, and actions. A routing layer that only knows API keys can cap spend and log traffic, but it cannot tell you whether a specific employee stayed inside the acceptable use policy.
Multi-model AI means using several AI models, often from different providers, and sending each task to the model best suited for it, such as a frontier model for complex reasoning and a small or self-hosted model for routine extraction. An AI gateway makes this manageable by giving callers one endpoint, one log, one spend report, and one set of data rules across every model.
An LLM router decides which model handles each request. It can route by static rule, by cost and latency, by the content or difficulty of the request, or by policy, such as keeping prompts that contain PII on a model inside your own tenant. It is usually a feature of an AI gateway rather than a separate product.
Applications and employees integrate with the gateway instead of with each model provider, so moving a task to a different model is a routing change rather than a rewrite. Logs, policies, and usage history stay with you, and you can switch models per task and pay for usage instead of committing every workflow to one vendor's seats.