AI Gateway has become a loose label. Some products come from API gateway or Kubernetes ingress backgrounds and have added AI features, such as Kong, Gloo, and Higress. Others started as AI-native tools, such as Portkey and OneAPI. Cloudflare AI Gateway belongs to the serverless public-cloud camp. Looking across these products, the common capabilities tend to fall into three groups:
- Ordinary API gateway features applied to LLM APIs: monitoring, logging, rate limiting, reverse proxying, request or response rewriting, and user-system integration. These mostly treat an LLM endpoint like any other API.
- API gateway features adapted to AI: token-based rate limits, prompt-based caching, prompt and response filtering, load balancing across multiple LLM API keys, and API translation between providers.
- New application-level features: embeddings and RAG exposed as APIs, vector and text database access, token-usage optimizations such as prompt simplification or semantic caching, and scoring LLM output.
Cloudflare AI Gateway is much simpler than the category name suggests. At its core, it is a reverse proxy. After trying it, I suspected I could assemble something similar with Cloudflare Workers without much trouble.
How it works
If an application already uses OpenAI's API, the main change is to replace the SDK base URL with https://gateway.ai.cloudflare.com/v1/${accountId}/${gatewayId}/openai. Traffic then enters and leaves through Cloudflare, which is how the platform provides monitoring, logging, caching, and related features.
That approach has clear advantages:
- Onboarding is easy. Change the base URL and the API format stays the same. The service is fully serverless, requires no server management, and is currently free, so basic monitoring data comes at no cost.
- Cloudflare's global network can provide some access acceleration. Compared with the latency of the LLM itself, that effect is probably small, with the clearest impact likely on time to first token.
- Routing through Cloudflare's network can also obscure the origin IP to some degree. In regions where OpenAI API access is restricted, this may help bypass those restrictions.
It also has drawbacks:
- All request information, including the API key, passes through Cloudflare, which creates security concerns.
- The gateway has no real plugin mechanism. Extending it is awkward and usually means adding another layer outside it.
- Because traffic comes from Cloudflare's global network, a key that keeps changing IP addresses could conceivably trigger blocking on OpenAI's side. That risk is not clearly documented.
Provider support
Cloudflare AI Gateway does not modify the LLM API; it only proxies requests. As a result, it can support most mainstream LLM APIs. The base URL simply points to the relevant provider, for example https://gateway.ai.cloudflare.com/v1/${accountId}/${gatewayId}/{provider}.
The one extra API it provides is called Universal Endpoint, which offers a basic fallback mechanism. A request can include queries for multiple providers, and if the first provider fails, the next one is called automatically.
Observability
Beyond basic QPS and error-rate dashboards, the monitoring view includes panels for tokens, cost, and cache hit rate, which are aimed at LLM workloads.
Logging feels similar to Worker logs: real-time logs are available, but historical logs cannot be queried. That is a weak point. Workers at least have third-party options for log persistence, while Gateway does not. It is possible to export real-time logs through an API and store them elsewhere, but that is cumbersome. Request and response logs are important for later optimization and even fine-tuning in many AI applications, so the lack of a built-in persistent logging option is a significant gap.
Caching
Caching is based on exact text matching, apparently implemented through Cloudflare Workers KV. A custom cache key can be set with cf-aig-cache-key, and users can configure cache TTL or bypass the cache. The current feature set does not appear to support semantic caching. Cloudflare says semantic caching will be added in the future.
Rate limiting
Rate limiting is still conventional QPS-based limiting. There is no token-based limiting for AI scenarios yet, which leaves room for improvement.
Custom metadata
Requests can include custom fields in headers, such as user information. These fields can then be searched in logs.
Assessment
Cloudflare AI Gateway's strength is simplicity. A user who has not previously used an AI gateway can connect in two or three minutes and get basic monitoring and caching. Cloudflare also has related AI services: Workers AI serves a large set of open-source models, and Workers provides edge compute. Combined, they can form a fully serverless AI system.
The main problem is that deeper features are limited, and extension is difficult. The practical workaround is to wrap the gateway with a Worker. At that point, Cloudflare might as well open-source AI Gateway as a template. Users could modify the code or write plugins for their own needs, and that could even grow into an ecosystem. After all, I strongly suspect the current AI Gateway is essentially a Worker template.