A 40-Call Audit Flagged 7 of 15 GPT Endpoints for Token Inflation

If you buy GPT access through anyone other than the model vendor itself, your per-token bill depends on generation settings you cannot see. A new audit found 7 of the 15 services it tested behaving as if someone had turned output length up. Researchers at the University of Science and Technology of China spent 40 API calls per service on 15 live LLM API services offering GPT-family models through OpenAI-compatible endpoints, and flagged seven for behaviour consistent with provider-side token inflation. The flags are signals, not proof of fraud, and the authors say so plainly. But the check is cheap enough to run before you sign and every quarter after, and the thing it looks for is invisible to both your quality evals and your invoice reconciliation.

The paper, The More It Says, the More You Pay, was submitted on September 17. Its most useful finding for a buyer is not the seven flags. It is that padded output barely hurts answer quality, and that the token count on the invoice is correct — the model really did generate every token you paid for. So the three things to do are unglamorous: measure output tokens per task against the first-party API, verify that your caps survive every hop, and write generation-config disclosure into the contract.

What Did the USTC Audit Actually Find?

Seven of the 15 services barely lengthened some of their answers when asked to, which is what a service that is already padding its output would do. The researchers sent 20 multiple-choice questions to each service, each question twice, for 40 calls per service and 600 in total. A service scoring 3 or more on their audit statistic was flagged; the seven flagged services scored 3 or 4. The services are anonymised as P01 to P15, identified only by the model name they advertise:

Advertised model	Services audited	Flagged	Which
gpt-4o-mini	4	0	—
gpt-5.4	3	2	P05, P12
gpt-5.4-mini	5	4	P06, P07, P11, P15
gpt-5.6-luna	2	1	P13
codex-auto-review	1	0	—

Source: Table 6 of arXiv 2609.20370.

Two patterns stand out. Every gpt-4o-mini endpoint scored zero, and every flag landed on a GPT-5-generation model name. And within each GPT-5 model name, at least one service came back clean: P13 was flagged while P14, advertising the same gpt-5.6-luna, scored zero. If the signal is real, it lives in the service, not in the model.

What you cannot do is look up your own vendor. The paper names none of them, and it does not say which of the 15 are resellers, aggregators or regional gateways. That is fine — you do not need their list. You need your own.

---

Why Padded Output Passes Your Evals and Your Invoice Check

Padding preserves the answer and the extra tokens really exist, so neither a quality eval nor a token-count reconciliation will catch it. A provider-side token inflation attack (PTIA) is an undisclosed provider-side manipulation that increases billable output tokens while the model still answers the question. The authors built five of them, at four points in the pipeline a provider controls: a hidden system-prompt instruction, a task-agnostic "overthinking" prefix on your query, a small model that elaborates your query before the real one sees it, a learned soft suffix on the input representation, and a LoRA fine-tune with a fixed trigger.

Each one pushed mean output to more than 10.2x the clean baseline, with ratios running up to 720.2x across the five attacks and four models. Accuracy hardly moved. Mean task accuracy was 77.9% clean against 79.8% under the hidden system-prompt attack, and the average drop across all attacks was 1.88 percentage points. The cheapest attack on the list — one extra instruction in a system prompt you never see — left the model slightly more accurate on the benchmark.

That is why your eval suite is the wrong instrument. It asks whether the answer is right. Padding keeps it right and makes it longer.

Your invoice reconciliation is the wrong instrument too, for a subtler reason. Most prior work on LLM overcharging attacks the count. An earlier tokenization-based overcharging method reached only 1.3x to 1.5x inflation. CoIn audits hidden reasoning-token counts and detects count inflation at up to 94.7% when run as a trusted third party. On the attack side, Hoque and colleagues showed in May that against three recent token-auditing frameworks, a provider with ordinary commercial capabilities could — in the most permissive setting — inflate hidden reasoning usage by 1,469% on average without detection — turning a $100 honest bill into roughly $1,569 — because every audit trusts some artifact the provider supplies.

PTIA is a different problem. The count is honest. The model really did say all of it. There are two questions a FinOps team has to ask about a token bill:

1. Did they bill me for the tokens they generated? That is reconciliation, and your gateway logs answer it.
2. Did they generate more tokens than my request needed? Nothing on the invoice answers that. Only a comparison does.

If you have been treating inference cost as a per-task number rather than a per-token rate, you already have the right frame. Output tokens per task is the metric that moves when someone pads.

How Does a 40-Call Probe Detect Padding Without a Reference Model?

It asks each question twice — once plain, once with a benign instruction that invites a longer answer — and measures how much longer the second answer gets; a service that is already padding has little headroom left. The authors call this PTIA saturation: an initial attack sharply lengthens output, but further strengthening or composition has much less effect. They trace it to stopping behaviour. The first inflation sharply lowers the probability of the end-of-sequence token; piling more on lowers it only marginally.

So the audit does not need to know what an honest answer looks like. It only needs to see whether the service still responds to a lengthening nudge. The protocol, per the paper's method section:

• 20 questions, each sent as two independent requests — original and probe-augmented — for 40 calls.
• The change in output length per question feeds two indicators: a non-positive change, and the size of any positive gain relative to the original length.
• The top three question scores are summed; a service at 3 or above is flagged.

On four open-weight models — Llama-3.1-8B, Ministral-3-14B, Qwen3-14B and Qwen3-32B — the audit reached an 85.1% average detection rate with false-positive rates below 2%. The false-positive rate was 0.02% with no system prompt, 1.83% with "You are a helpful assistant", and 0.73% with a safety prompt. It beat a baseline that needs a trusted local copy of the model (71.7%) and one that needs historical clean responses (66.7%).

Evasion is the obvious objection. The authors concede that an audit-aware provider may try to withhold inflation whenever it recognises the probe, and argue this is hard because the plain and probed requests go out separately.

Two practical limits. First, the code is not out: the authors say they plan to release an artifact with the audit code and configurations, so today you replicate the protocol rather than download it. Second, 20 questions is a small sample, and the paper says so — larger audit sets give stronger statistical evidence at additional query and token cost. If you have read our piece on how few model gaps survive a confidence interval, you know what a 20-item test can and cannot tell you.

---

The Strongest Case Against Acting on These Flags

The flags cannot establish intent, and several honest explanations fit the same data. The authors are explicit: without backend ground truth, its alerts cannot by themselves establish deliberate provider manipulation. The steel-man goes further than that.

• The paper reports no first-party control. The real-world audit covers 15 services offering GPT-family models through OpenAI-compatible endpoints. The paper reports no score for OpenAI's own API alongside them, so you cannot see how the vendor's endpoint scores on the same 20 questions.
• The detector was calibrated on different models. Its accuracy numbers come from four open-weight models. Every flag landed on a GPT-5-generation name, and no gpt-4o-mini endpoint was flagged. Either padding concentrates on the newer models, or the probe reads them differently from the models it was tuned on. The paper cannot separate the two.
• Honest deviation is normal. A July audit called IRIS flagged 14 of 15 same-model provider pairs — and attributed the differences to genuine quantization and kernel deviations. Two providers serving "the same model" routinely do not serve the same thing.
• A gateway can inflate your bill by accident. An issue opened September 14 against OrcaRouter Lite, an open-source router, reports that its request schema silently strips max_completion_tokens, the field newer OpenAI SDKs send. A request capped at 10 tokens came back with several hundred. Nobody padded anything on purpose; the cap just never arrived.

Here is why none of that lets you off. You do not need to prove intent to act on cost. A service whose answers run long — by design, by a wrapper prompt nobody documented, or by a dropped parameter — costs you the same either way. The flag is a reason to measure, not a reason to accuse.

Where Does This Risk Actually Sit in Your Stack?

Anywhere your code points an OpenAI-compatible client at a base URL that is not the model vendor's own — and there are more of those than your procurement records show. The OpenAI-compatible API became the lingua franca, which means any intermediary can sit behind the same client library with a one-line change. Three kinds deserve attention.

Aggregators. OpenRouter, by default, load-balances across providers weighted by the inverse square of the price, with fallbacks on unless you turn them off. That is sensible for availability, but it means the provider serving request 1 may not serve request 2 — so an audit of "OpenRouter's gpt-5.4" is an audit of a mix, drawn from what Stripe's announcement describes as 400+ models from more than 80 providers. (Stripe agreed to acquire OpenRouter in August; see what that means for your contract.)

Commercial gateways. An April study, GateScope, measured 10 commercial LLM API gateways and found silent model substitutions and deviations from announced pricing. One platform showed a 62.8% billing gap even though its token usage was similar to the others; another's gpt-5 responses were consistent with the claimed model 13.09% of the time, against a 97.09% baseline. Model substitution is the sibling risk to padding — the same one we covered in CISA's distillation advisory and silent model downgrades.

Shadow APIs. A March audit identified 17 shadow APIs used in 187 academic papers — third-party services claiming official model access without regional limits — and found identity-verification failures in 45.83% of fingerprint tests. These are the endpoints someone on your team signed up for on a card because the official API needed a purchase order, or was not available in their region.

Your own proxy is in scope too. A self-hosted LiteLLM or similar gateway is a place where a default system prompt or a stripped parameter can lengthen every response in the company. The gateway you chose for cost control is also the one place you can measure this cheaply, because every request already passes through it.

What to Do Before You Route Another Dollar Through a Gateway

Treat output tokens per task as a vendor metric, measure it against the first-party API, and make every intermediary disclose what sits between your request and the model.

This Week:

1. List every base URL. Search your code and config for OpenAI-compatible base URL overrides, then cross-check expense reports for card-billed API credit. Anything that is not the model vendor's own endpoint or a cloud contract you negotiated goes on the audit list.
2. Pull 30 days of output tokens per task, by endpoint. Use your gateway logs or a tracing tool such as Langfuse, grouped by prompt template. Report completion_tokens and, where exposed, reasoning tokens. Spend moves with volume; output tokens per task moves with padding.
3. Check that your caps survive the hop. OpenAI defines max_completion_tokens as an upper bound on generated tokens, including visible output and reasoning tokens. Send a 10-token cap through every gateway and read the usage block. If hundreds come back, you have found the OrcaRouter bug in your own stack.

This Month:

4. Run a paired baseline against the first-party API. Fix a set of at least 20 questions drawn from your real traffic, send them with identical parameters to the vendor's API and to each third-party endpoint serving the same model name — say, GPT-5.4 — and compare output-length distributions. On an aggregator, pin the provider (OpenRouter's only list with allow_fallbacks set to false) or you are auditing a lottery. An endpoint that runs materially longer on identical prompts gets a ticket to its account manager.
5. Add the paper's two-request length probe for any endpoint where no first-party baseline exists. Treat it as a screen, not a verdict, until the authors' artifact ships and you can match their scoring exactly.

Before Renewal:

6. Write generation-config disclosure into the contract. The provider lists every system prompt, prefix, query rewrite, adapter or decoding change applied to your traffic; gives notice before changing any of them; grants you the right to run length and identity audits; and credits you if output length on an agreed test set drifts outside an agreed band from the first-party API. The paper's authors call for exactly this: provider-side disclosure or verifiable attestation of generation configurations.
7. Put the probe on a calendar — quarterly, and whenever the advertised model name changes. The authors' own advice is to audit before adoption and repeat periodically. A one-time check at onboarding tells you what the service did during onboarding.

---

The Bigger Picture

Pay-per-unit markets where the seller counts the units tend to end up needing evidence the seller does not control, and programmatic advertising learned that at a cost of billions. When the ANA examined log-level data from 21 major advertisers, it found that only 36 cents of every dollar entering a demand-side platform effectively reached the consumer, and put the efficiency gains available to marketers at $22 billion. It took the advertisers' own log-level data to see it.

Tokens are the new impression. The unit is invisible, the seller counts it, and a growing layer of resellers and gateways sits between the buyer and the thing being bought. Hoque and colleagues conclude that honest billing will need verification tied to evidence the provider does not control — trusted execution attestation, cryptographic proofs of inference, or third-party re-execution. Until an intermediary offers you one of those, your own measurements against the first-party API are the only evidence in the room the seller did not produce.

Your invoice will be accurate. That was never the question.

Continue Reading

• Inference Cost per Million Tokens: Price the Task, Not the Rate
• CISA Named 6 Distillers. Your Agent Fleet Fits the Spec.
• Best LLM Gateways for Cost Control: Self-Host First
• AI Gateway vs API Management: No Managed Gateway Caps Your Spend
• Model Router Buyer's Guide: Buy Failover, Not Judgment
• Agentic AI Pricing: Don't Buy Consumption Without a Cap