Your API gateway can already count LLM tokens. What none of the managed platforms can do is stop a runaway agent before the bill arrives — and that gap, not the feature checklist, is the thing that should decide this purchase.
The question lands on platform teams in the same form every time. Four product teams are calling three model providers through four different SDKs, finance wants a per-team number, security wants the prompts logged, and somebody has forwarded a vendor deck for an AI gateway. Meanwhile there is a perfectly good API management platform already in production with a team that knows how to operate it. Do you buy the new thing or extend the old one?
The honest answer depends almost entirely on which API management platform you already run, and the spread between the best and worst answer is roughly 5x on price for near-identical capability.
The Short Answer at Five Million Calls a Month
If you already run Azure API Management, put LLM traffic behind it and stop shopping. If you run Apigee, do not buy your way up to a Comprehensive environment for this — put a lightweight LLM gateway behind your existing proxy. If you run Kong, the open-source AI plugins cover routing and prompt guards, but the token limiter you actually want is Enterprise-licensed and metered per model.
Everything below is normalised to one workload so the numbers are comparable: 5 million LLM calls per month, roughly 1,800 tokens each (1,400 in, 400 out), 4 models across 3 providers, 12 consuming teams, one region, peak 200 requests/second. Vendors each prefer a different denominator; this is the same denominator for all of them. All prices checked 26 August 2026.
| Option | Monthly cost at this workload | Token-aware limits | Semantic caching | Who it is for |
|---|---|---|---|---|
| Azure API Management (Standard v2) | ~$700 (one unit; 5M calls sit inside the 50M included) | Native llm-token-limit policy, per-minute and per-quota |
Yes, needs an external RediSearch cache (+$12–$158) | You already run APIM and your identities are in Entra |
| Kong Konnect Plus + AI Gateway | ~$1,400 ($200 control plane + $800 request overage + $400 for 4 models) | Yes, but Enterprise-tier plugin only | Yes, Enterprise-tier plugin, you supply Redis or pgvector | You already run Kong and want one gateway for REST and LLM |
| Google Apigee (pay-as-you-go) | ~$3,931 ($3,431 Comprehensive env + $500 extensible calls) | Token quotas via AI policies | Yes, but Comprehensive environment only, plus Vertex AI Vector Search | You are already on a Comprehensive environment for other reasons |
| LiteLLM (self-hosted) | $0 licence + your own compute | Yes, virtual keys, budgets, TPM/RPM limits | Via your own cache layer | You have a platform team and want the cheapest hard-to-beat baseline |
| Cloudflare AI Gateway | $0–$5 (Workers Paid) | Rate limiting, no per-team token budgets | Yes, included free | Fast start, low governance requirements |
The loser in this comparison is Apigee — not as an API management platform, but for this specific job. You pay a $3,431/month environment premium to unlock the one feature whose value you cannot know until after you have measured your own traffic.
What Token-Aware Rate Limiting Actually Enforces
Token-aware rate limiting is a soft ceiling with documented overshoot, not a spending cap. Every product in this comparison describes it as a budget control. Read the vendor's own limitations page and it stops being one.
Azure's llm-token-limit policy is the most honest documentation of the three, and it is worth reading before you write the business case. Microsoft states plainly that the policy tracks token usage independently at each gateway where it is applied, including workspace gateways and regional gateways in a multi-region deployment, and that "it doesn't aggregate token counts across the entire instance." A 500,000-token monthly quota deployed across three regional gateways is a 1.5-million-token quota.
The same page documents three more edges that matter more than any feature bullet:
- Streaming defeats exact accounting. "When streaming is enabled in the API request (
stream: true), prompt tokens are always estimated regardless of theestimate-prompt-tokenssetting. Completion tokens are also estimated when responses are streamed." Every chat UI you ship streams. - Images are overcounted. With streaming on or estimation enabled, the policy "overcounts each image as a maximum of 1200 tokens." Multimodal workloads will be throttled early.
- Concurrency overshoots. "Concurrent or near-concurrent requests can temporarily exceed the configured token limit." At 200 requests per second, "temporarily" is a meaningful number of dollars.
Kong is in the same position and says so more briefly. Its AI Rate Limiting Advanced plugin supports counting by total_tokens, prompt_tokens, completion_tokens or cost, which is genuinely useful — the cost strategy computes (prompt_tokens × input_cost + completion_tokens × output_cost) / 1,000,000, so you can express a limit in dollars rather than tokens. The catch is in the same document: "The cost for the AI Proxy or AI Proxy Advanced is only reflected during the next request." Enforcement lags one request behind actual spend, per consumer.
Apigee reaches the same place by a different route, with token limits implemented through reference solutions Google publishes as samples rather than a single first-party policy. AWS is the newest entrant and the most direct about scope: AgentCore gateway, announced 6 August 2026, supports request (RPS/RPM), token (TPM) and connection (CPS) limits, with token limits applying "to inference targets only," keyed by qualifiedModelId, JWT claims or IAM principal.
On the managed platforms, a token limit is an alerting and throttling primitive. If you need a genuine hard stop — the kind finance means when they say "cap it at $40,000" — none of Azure, Kong or Apigee will give you one. You build that on the metric emission, not the limiter, and you accept that the cap trips after the money is spent.
The exception is self-hosted, and it is worth knowing before you sign anything: LiteLLM's proxy documents budget reservation, which estimates a request's maximum cost from the request body, reserves that amount against the key, team or user budget, and — in its own words — "rejects the request before sending it to the provider" if the reservation would exceed the budget. That is a pre-spend ceiling rather than a post-hoc one, and it is available only on the option in this comparison with no vendor SLA behind it.
Semantic Caching Is Workload-Dependent and Priced as If It Weren't
Published hit rates for semantic caching range from under 4% to over 90%, and the variable is how repetitive your traffic is. The pitch is that near-duplicate prompts get served from cache and never reach the model. That is real. What no vendor can tell you is the number it will produce on your prompts — and this is the capability driving the largest price differences between these platforms.
At the pessimistic end, SCALM, an academic study of semantic caching for LLM chat services, evaluated against the MOSS and LMSYS conversation datasets and found baseline cache hit ratios of 3.8% on MOSS and 6.4% on LMSYS, with its own improved clustering approach reaching an 11.7% hit ratio and an 8.4% token saving ratio. Read the configuration before you quote that at a vendor: the caches tested hold 20 to 200 entries, and both datasets are open-domain multi-turn chat — brainstorming, writing, complex instructions — which is close to the least repetitive traffic a cache can be handed.
At the optimistic end sits AWS's own benchmark, which streamed 63,796 real Chatbot Arena queries and paraphrased variants through ElastiCache as a semantic cache and reports a 23.5% hit ratio at a 0.99 similarity threshold, 56.0% at 0.95 and 90.3% at 0.75, with cached-response accuracy holding between 91% and 92% across that whole range. Discount it twice — AWS sells the cache, and a dataset seeded with deliberate paraphrases is friendlier than organic traffic — and it still lands nowhere near single digits.
A third measurement explains both. Category-aware caching research finds that "high-repetition categories achieve 40-60% hit rates while low-repetition or volatile categories achieve 5-15%," because code and documentation queries cluster densely in embedding space while conversational ones do not. Your hit rate is a property of your traffic mix, not of the gateway. None of these three studies predicts it for you.
Then there is the failure mode, which Kong documents in its own product page. From the AI Semantic Cache plugin docs: "When Exact Caching is enabled, the AI Semantic Cache plugin may still return results for queries that are similar but not identical. This is expected behavior." Production teams hit exactly this — "what is the return policy?" and "what is the exchange policy?" are semantically adjacent and operationally different, and a threshold loose enough to produce a useful hit rate is loose enough to answer one with the other.
Now price it. Kong's plugin is Enterprise-tier and requires you to run Redis with vector similarity search, Valkey, or PostgreSQL with pgvector. Azure requires Azure Managed Redis or another external RediSearch-compatible cache — $0.016/hour for a Balanced B0 up to $0.216/hour for a Memory Optimized M10, so $12 to $158 a month.
Apigee is the expensive one. Google's own tutorial states that semantic caching policies can only be deployed in Comprehensive environments, and the SemanticCacheLookup policy additionally requires a Vertex AI project, the Text embeddings API, and a Vector Search index with a deployed endpoint. A Comprehensive environment is $3,431 per month per region against $365 for a Base environment, per Apigee's published pay-as-you-go examples — a $36,792 annual delta, before the always-on Vector Search endpoint, to buy a hit rate nobody has measured on your traffic.
Take semantic caching out of the platform decision and measure it separately. Replay a week of production prompts against an open-source cache at two or three similarity thresholds; that costs days and produces the only number that matters. Buying a tier upgrade before that measurement is paying a platform premium for a figure that could plausibly be 5% or 50%. If prompt-level cost reduction is the goal, provider-side prompt caching does more for less, which is a trade-off worth understanding before you route around it.
Where the Money Actually Goes at This Volume
Azure API Management is the cheapest incumbent path by a wide margin, because it charges for gateway capacity rather than per model or per million requests. The pricing models are structurally different, and that structure — not the rate card — is what produces the 5x spread.
Azure. A Standard v2 unit is $0.9589 per hour in East US, or $700 a month, and it includes 50 million API requests; calls beyond that are $0.025 per 10,000. Our 5 million calls are a tenth of the allowance, so they cost nothing extra — the $700 is the whole bill. Basic v2 is $0.20548/hour ($150/month) and Premium v2 is $3.83562/hour ($2,800/month), all per Microsoft's retail price list. The AI gateway is not a separate SKU; Microsoft is explicit that it "extends API Management's existing API gateway; it's not a separate offering." That is the whole reason Azure wins on cost here.
Kong. Konnect Plus bills per control plane and per request. Per Kong's pricing page: $200/month for a hybrid gateway control plane, 1 million API requests included, $200 per additional million, and AI Gateway model proxy at $100 per model per month up to 5 models. Our workload — 5 million requests, 4 models — lands at $1,400/month, and the per-request line grows linearly while Azure's does not. At 10 million requests you are at $2,400 and at the Plus ceiling.
Apigee. $3,431/month for the Comprehensive environment, and calls priced by proxy classification: $0.00002 per call for Standard proxies, $0.0001 for Extensible — a 5x multiplier. At 5 million extensible calls that is $500, for $3,931 all in. Stay on a Base environment with Standard proxies and you are at roughly $465, but without semantic caching.
The open-source floor. LiteLLM is free to self-host with virtual keys, budgets, teams, load balancing and TPM/RPM limits in the open-source build; Enterprise adds SSO, JWT auth and audit logs at quote-only pricing. Portkey is $49/month for 100,000 recorded logs with $9 per additional 100,000 — around $490/month at our volume, and free for 10,000 logs. Cloudflare AI Gateway gives away the core: "AI Gateway's core features available today are offered for free," with logs capped at 100,000 total on Workers Free and 10,000,000 per gateway on Workers Paid.
The pattern is clear enough to act on. Capacity-based pricing (Azure) is flat as token volume grows. Per-request and per-model pricing (Kong, Apigee) scales with exactly the thing that is about to grow fastest in your estate. Model your 18-month volume, not today's.
Logging, PII and the Compliance Artifact Nobody Specs
Prompt and completion logging is the requirement that most often decides this, and it is the one least likely to be in the RFP. Under the EU AI Act's record-keeping obligations for high-risk systems, the log is not a debugging convenience — it is evidence. Where it lands, how long it is kept, and who can read it are procurement questions.
Azure logs prompts and completions to Azure Monitor and emits per-consumer token metrics through the llm-emit-token-metric policy, with custom dimensions for client IP, API ID or a user header, feeding a built-in analytics workbook. It also applies Azure AI Content Safety moderation to prompts in-flight. The data stays in your subscription, which is the part compliance cares about.
Kong's answer is the AI PII Sanitizer plugin, which "protects sensitive information in client request or response bodies before they reach upstream services or clients," alongside AI Prompt Guard for allow/deny expressions and AI Semantic Prompt Guard for topic-level lists. Kong also brokers third-party guardrails — Azure Content Safety, AWS Guardrails, GCP Model Armor and Lakera Guard all appear as plugins, which is a genuine advantage if you have already bought one of those.
Apigee routes safety through Model Armor, integrated as Apigee policies that filter prompt injection and jailbreak attempts, and Google's guidance is that you "safely log any LLM interaction with Cloud Logging, de-identify it, and inspect it from a familiar interface."
The trap is redaction placement. If PII is stripped at the gateway, your log is compliant and your debugging is degraded — you cannot reproduce the prompt that produced the bad answer. If it is stripped after logging, you have a retained plaintext copy of everything your users typed. Decide which, in writing, before you pick a product, because the logging gap is what turns a shadow-AI problem into a disclosure event.
Failover and Routing Are Table Stakes Now
Every platform here does model failover competently, so it should carry almost no weight in your decision. This was a differentiator eighteen months ago. It is not one now.
Azure exposes backend load balancing with round-robin, weighted, priority-based and session-aware strategies, plus a circuit breaker with dynamic trip duration that honours the backend's Retry-After header — which matters specifically for keeping Provisioned Throughput Unit capacity saturated before spilling to pay-as-you-go. Kong's AI Proxy Advanced plugin load-balances across providers, and the base AI Proxy plugin already speaks OpenAI, Azure OpenAI, Amazon Bedrock, Anthropic, Gemini, Vertex AI, Cohere, Mistral, Hugging Face, Llama, xAI, DashScope, Cerebras, DeepSeek, Ollama, Databricks and vLLM. Apigee publishes routing and circuit-breaking as reference implementations.
What is worth weighting is where these capabilities sit on the licence boundary. Kong released six AI plugins as free and open source in Gateway 3.6 — AI Proxy, AI Request Transformer, AI Response Transformer, AI Prompt Guard, AI Prompt Template and AI Prompt Decorator, "entirely free and open source for everyone." The ones that govern cost — AI Proxy Advanced, AI Rate Limiting Advanced, AI Semantic Cache — are Enterprise-tier. That is a reasonable commercial split, and it means a Kong proof of concept that runs beautifully on the free plugins tells you nothing about the price of the production configuration.
Routing is also where multi-provider strategy stops being theoretical. A Harris Poll survey of 600 CIOs at companies above $500M revenue across eight countries, fielded December 2025 to January 2026 and commissioned by Dataiku — which sells a multi-provider abstraction layer, so weigh the direction of the finding accordingly — found 81% expect to rely on two or more LLM providers in 2026, 93% say different models perform better for different use cases, and 55% have already switched at least once, with cost as the main driver. If you are single-provider today, price the gateway as if you will not be. Kong AI Gateway, Azure AI Foundry and OpenRouter all assume you will not be.
Who Should Not Buy Each of These
This is the section vendor comparisons skip, and it is the one that saves money.
Do not put LLM traffic behind Azure API Management if your models are mostly outside Azure and your consumers do not authenticate through Entra — the counter-key model is at its best keyed to a subscription key or Entra identity, and loses most of its value otherwise. Also skip it if you need a budget ceiling that holds across regions; Microsoft's own documentation says counters do not aggregate across the instance.
Do not buy Kong for this if you are on Konnect Plus and heading past 10 million requests a month or more than 5 models, because you will be renegotiating into Enterprise quote-only pricing inside a year. Skip it too if the token limiter is the whole reason you are buying — that plugin is Enterprise-tier, so the free-plugin evaluation is not the product you will pay for.
Do not buy Apigee for this if LLM traffic is the reason. A Comprehensive environment is $3,431/month, semantic caching is the only capability it uniquely unlocks here, and semantic caching is the capability whose payoff is least knowable in advance. If you already run Apigee for your REST estate, keep it there and put an LLM-native gateway behind it.
Do not self-host LiteLLM if you have no platform team to page at 3am. It is now on the data path for every AI feature you ship, and "free" is a licence statement, not an operational one. Skip it too if procurement requires a vendor SLA on that path.
Do not use Cloudflare AI Gateway if you need per-team token budgets or an audit-grade log. The free core is genuinely free and the setup is minutes, but 100,000 logs on the free plan does not survive contact with a real workload, and the data path is Cloudflare-hosted — which is a data-residency conversation, not a technical one.
Five Questions That Predict Regret
Answer these before you look at a single feature matrix. In practice they determine the outcome more reliably than any capability comparison.
- Where does identity live? The rate limiter is only as good as its counter key. If your consumers already carry an Entra token, an IAM principal or a JWT claim your gateway can read, use the gateway that reads it natively. Everything else is glue you maintain.
- Do you need a hard cap or a soft cap? No managed API management platform sells a hard cap; the only pre-spend ceiling in this comparison is self-hosted LiteLLM's budget reservation. If finance believes an incumbent gateway bought them one, correct that before the first invoice, and build the real stop on emitted metrics plus an automated key revocation.
- Do you stream? If yes, token accounting is estimated, and your chargeback numbers are approximate by design. Decide now whether approximate is acceptable, because it will not become exact later.
- Who owns the prompt log, and for how long? Answer as a retention schedule and an access list, not a product name. This is the requirement most likely to eliminate an option outright.
- How fast does the model list change? Kong charges per model. Apigee charges per call and reclassifies proxies. Azure charges for capacity. Pick the pricing axis that grows slowest for you, then verify it against an 18-month volume forecast rather than this quarter's.
There is one more, and it is the cheapest test available: run the workload through a self-hosted open-source gateway for two weeks first. You will discover your actual token distribution, your actual streaming ratio and your actual per-team split — and those three numbers are what every quote you receive will be wrong about.
The Bottom Line
The AI gateway category is doing what the API management category did in 2014: selling a control plane before anyone has agreed what needs controlling. The capabilities are real, they are converging fast, and within two product cycles token-aware limits and prompt logging will be checkbox features on every gateway — Azure has already folded them into its existing gateway rather than shipping a separate SKU, and AWS added token limits to AgentCore this month.
What will still be true in two years is the structure. Capacity pricing beats per-request pricing when the thing being counted is about to grow 10x. Managed soft caps stay soft. Streaming stays estimated. Logs stay a compliance artifact, not a debugging one. And the gap between the platform your team can operate and the platform with the best feature matrix will keep costing more than the licence.
Extend what you already run, unless what you already run is Apigee. Then put something small in front of the models and keep your Comprehensive environment budget for something that earns it.
Buy the ceiling you can actually enforce, not the one on the slide.
Continue Reading
- Best LLM Gateways for Cost Control: Self-Host First
- Your AI Router Is Trading a 10x Discount for a 2.5x One
- Stripe Bought OpenRouter. A Toggle Is Not a Contract.
- Meta's Agent Model Fits on a Laptop. Nothing Logs It.
- Copilot Retried Into GitHub's Outage. Cap Your Agents.
- AI Agents Have Admin Access. 57% of Firms Aren't Ready.
- Nutanix AI Gateway: Why Token Bills Are About to Explode
- EU AI Act Governance Tools: Buy Inventory, Not Policy Packs