AI Gateway

An AI gateway is a proxy that sits between applications and the AI models they call. Applications send every request to one endpoint, and the gateway handles provider credentials, chooses the model, retries failures, enforces usage limits, serves cached answers, screens content, and records each call. It is the AI-specific form of an API gateway: large language model (LLM) requests are priced and limited by tokens, responses often stream, payload formats differ by provider, and the prompts and answers themselves need inspection for prompt injection and leaked personal data. Hosted gateways such as Cloudflare AI Gateway, Vercel AI Gateway, and OpenRouter, and self-hosted proxies such as LiteLLM, Kong AI Gateway, and Envoy AI Gateway, typically expose an OpenAI-compatible API, so existing client code only changes its base URL.

Core Functions#

Most gateways combine the same building blocks:

• Unified API and keys: One request format covers many providers. The gateway stores the provider credentials and issues its own keys to each team or application, so access can be revoked or metered without touching provider accounts.
• Model routing and fallbacks: Each request goes to a model chosen by rules, cost, or a learned router, and a failed or rate-limited call is retried on a backup model or provider.
• Token budgets: Limits count tokens per minute and spend per period. Azure API Management's llm-token-limit policy returns 429 when a tokens-per-minute limit is exceeded and 403 when a quota runs out, and concurrent requests can briefly overshoot because a call's token cost is known only when it finishes.
• Semantic caching: Exact-match and meaning-based caches return stored answers for repeated or equivalent prompts without calling a model.
• Guardrails: Prompts and responses are scanned for injected instructions, personal data, and policy violations before they pass through.
• Observability: Every call is logged with its model, token counts, latency, and cost. The OpenTelemetry GenAI semantic conventions standardize attribute names such as gen_ai.usage.input_tokens, so these logs work across monitoring tools.

A request passes these checks in order:

Gateways for Self-Hosted Models#

Teams that run their own models need a gateway that understands model servers. The Kubernetes Gateway API Inference Extension turns gateways such as Envoy Gateway and GKE Gateway into inference gateways: an endpoint picker reads metrics from each model server replica, such as prefix cache status and which LoRA adapters are loaded, and sends each request to the replica that can serve it fastest. This keeps KV cache reuse high on servers such as vLLM, which a round-robin load balancer cannot do because it ignores what each replica has cached.

The same gateway controls apply to computer vision APIs. Ultralytics Platform puts them in front of its shared model API: requests authenticate with any active workspace API key, each key is limited to 20 requests per minute, and throttled calls receive a 429 response with a Retry-After header. Requests sent straight to a dedicated endpoint bypass that limiter.

AI Gateway vs. Related Terms#

An API gateway handles authentication, rate limits, and routing for any web service and counts requests. An AI gateway adds token accounting, provider translation, streaming support, and content inspection. Model serving runs the model and returns predictions, and the gateway sits in front of one or more serving endpoints. AI agent orchestration plans multi-step work across tools and models, and the agent's individual model calls can pass through a gateway. The Model Context Protocol connects agents to tools, and some gateways, including Azure API Management, govern MCP servers with the same keys, limits, and logs.

Gateways add one network hop to every call and become a dependency that every AI feature shares, so production deployments run them redundantly across zones. Hosted gateways also see every prompt and response, which brings them into the scope of data privacy reviews, and gateways such as Cloudflare AI Gateway and LiteLLM can turn off the storage of prompt and response bodies.