Fine-tuning is cheap to do and expensive to keep. For a typical enterprise knowledge workload — 40,000 documents, 50,000 queries a month — the training run costs about $30, and the decision of where to serve the result swings the monthly bill from $326 to $28,908. Your data and your traffic are identical in both cases. What differs is the serving contract the platform makes you sign.
Start with RAG. Keep starting with RAG. Fine-tune only when you are buying behaviour — format discipline, tool-calling, refusal, house style, reading imperfect retrieval — and never when you are trying to buy facts. Then, before you approve the training job, price the serving line, because that is the line that recurs.
Prices below were fetched from each vendor's live pricing page on 10 September 2026.
| Path | Training cost | What you pay to serve | ~50K queries/mo | Cost while idle |
|---|---|---|---|---|
| RAG on base model (Gemini 3.5 Flash) | none | base per-token rate | $680–$740 | $50–$110 (vector DB) |
| Vertex AI tuned endpoint | per training token | 1.5× base per-token rate | $326 | $0 |
| Fireworks AI LoRA | $0.50–$10 / 1M tokens | base per-token rate, no premium | scales with tokens | $0 |
| Together AI LoRA | $0.48–$2.90 / 1M tokens | serverless, or $3.99–$8.99/GPU-hr dedicated | varies | $0 serverless |
| Azure OpenAI fine-tune | per training token | per-token + $1.70/hr hosting | $1,241 in hosting alone | ~$1,241/mo |
| Bedrock via Provisioned Throughput | $1.49 / 1M tokens + $1.95/mo storage | committed capacity, $21.18–$39.60/hr | $15,461–$28,908 | identical — it is a rental |
| Bedrock custom on-demand (Nova, Llama 3.3 70B) | same | base per-token rate, no premium | scales with tokens | $0 |
| Bedrock Custom Model Import | train elsewhere | $0.05718 per CMU-minute; 8 CMUs for a 70B | ~$27.45/hr per running copy | $0 when no copy runs |
| OpenAI direct | $1.50–$25 / 1M tokens | fine-tuned per-token rate | closed to new customers | — |
The gap between the cheapest and dearest way to keep a customized model in production is roughly 89× at this volume. The gap in training cost, across the same platforms, is a rounding error. Most fine-tuning business cases are built on the second number and never model the first.
One of the Two Big Fine-Tuning Doors Just Closed
If your production model is a current frontier model, fine-tuning may not be available to you at any price. This is a supply constraint, not a cost one, and it is new.
OpenAI is winding down its self-serve fine-tuning platform. Per OpenAI's own deprecations page, from 7 May 2026 organizations that had not previously run fine-tuning could no longer create training jobs; from 2 July 2026 the restriction extended to organizations that had not run inference on a fine-tuned model in the prior 60 days; and on 6 January 2027 even active existing customers lose the ability to create new jobs. Inference on already-tuned models survives only "when the underlying base model is deprecated" — that is, until it doesn't. The models still listed on OpenAI's fine-tuning price table are gpt-4.1 and gpt-4o at $25 per 1M training tokens, gpt-4.1-mini at $5, gpt-4o-mini at $3, gpt-4.1-nano at $1.50, and o4-mini billed at $100 an hour rather than per token. There is no fine-tuning row for the current flagship.
Anthropic's position is narrower still. The only supervised fine-tuning path for a Claude model is Claude 3 Haiku on Amazon Bedrock, generally available since November 2024 in US West (Oregon) — one model, one region, two generations behind the Opus 5 and Sonnet 5 rates most enterprises are actually running.
So for a large share of buyers the question answers itself: you cannot fine-tune the model you have shipped. What remains is open-weight models on Together, Fireworks, Databricks or Bedrock Custom Model Import, Gemini on Vertex AI, and the GPT-4.1 and GPT-4o families through Azure OpenAI — which, unlike OpenAI direct, still takes new fine-tuning customers and lists GPT-5 as fine-tunable by application. That is the real menu, and it is where the pricing below applies.
The Workload This Article Prices
Every vendor quotes the metric that flatters it, so here is one workload, held fixed:
- Corpus: 40,000 documents averaging 600 tokens — 24M tokens indexed.
- Volume: 50,000 queries per month, roughly 1,650 a day.
- RAG prompt: 1,500 tokens of stable system prompt and few-shot examples, 4,200 tokens of retrieved context (six chunks at 700), 300 tokens of question — 6,000 input tokens.
- Fine-tuned prompt: 200 tokens of residual instruction plus the same 300-token question — 500 input tokens.
- Output: 400 tokens either way. Fine-tuning shortens the prompt; it does not shorten the answer.
- Model: Gemini 3.5 Flash, which Google prices at $1.50 per 1M input tokens and $9.00 per 1M output.
Google Vertex AI is the only major platform that publishes both a base rate and an explicit tuned-model premium, which makes it the one place you can hold the model constant and change only the customization. Google's Vertex AI pricing page states that from Gemini 3 onward, a tuned model endpoint's prediction price is 1.5 times the base model's; older Gemini 2.5-era tuned models stayed at base rates. CloudZero's Vertex pricing review, updated 4 September 2026, reads the same line the same way.
RAG, monthly: 300M input tokens at $1.50 = $450, plus 20M output at $9.00 = $180, so $630 of inference. Embedding the corpus once at Gemini Embedding's $0.15 per 1M is $3.60; query embeddings are $2.25 a month. Add the index: Pinecone's Standard plan carries a $50/month minimum usage commitment on top of $0.33/GB/month storage and $16–$18 per million read units, while a pgvector-capable Postgres on Supabase's Large compute add-on is $110/month. Total: $680 to $740.
Tuned endpoint, monthly: 25M input tokens at $2.25 = $56.25, plus 20M output at $13.50 = $270. Total: $326.
Fine-tuning saves $354 a month, or $0.0071 per query, on this workload. Hold that number. Everything else in the business case has to fit inside it.
The Serving Contract Is the Whole Decision
A fine-tuned model is not a file you own — it is a tenancy, and the tenancies differ by two orders of magnitude. Four shapes exist, and they are what actually determine whether fine-tuning is cheaper than retrieval.
Multi-tenant adapter serving, no premium. Fireworks AI states it plainly on its pricing page: "Serve fine-tuned models for the same price as base models," with LoRA supervised fine-tuning at $0.50 per 1M training tokens for models up to 16B, rising to $10.00 above 300B. There is no hosting fee and no idle charge. On this contract, fine-tuning's token math is unconditionally favourable — the prompt gets shorter and the rate does not move.
Multi-tenant with a premium. Vertex AI's 1.5× multiplier. Cheaper than RAG here, but conditionally, and the condition is a trap (next section).
Dedicated capacity, billed hourly. Azure OpenAI, now served through Azure AI Foundry, is explicit: Microsoft's documentation for deploying a fine-tuned model states that each deployed custom model "incurs an hourly hosting cost regardless of whether chat completions or response API calls are made to the model," and that a deployment idle for more than 15 days is deleted outright. Microsoft's own support answer put that charge at $1.70/hour for both GPT-4o and GPT-4o-mini deployments — $1,241 a month across a 730-hour month, before a single token moves. Azure does offer a Developer tier with no hourly hosting fee, but it carries no availability SLA and Microsoft says it is "designed for model candidate evaluation and not for production use." Use it for the bake-off; do not ship on it.
Rented throughput, committed. Amazon Bedrock is the extreme — on its older path, and only there. AWS's Provisioned Throughput page still states flatly: "If you customized a model, you must purchase Provisioned Throughput to be able to use it." That sentence is no longer universally true, and the exception is worth more to a buyer than the rule (below). Where it does apply, the same page offers no-commitment, one-month and six-month terms, and says of the committed ones that you "can't delete the Provisioned Throughput" until the term is over — no-commitment capacity is available for a restricted set of models, and it is billed at a higher hourly rate. Bedrock's pricing page shows those rates running from $21.18/hour for a Llama 2 13B model unit on a one-month commit to $39.60/hour for Cohere Command — $15,461 to $28,908 a month, whether you serve 50,000 queries or zero. Against RAG's $680, that is 23× to 43× for the same answers.
There are two escape hatches, and buyers should know both. First, AWS separately documents deploying a custom model for on-demand inference: "With on-demand inference, you only pay for what you use and you don't need to set up provisioned compute resources." This covers models you fine-tuned on Bedrock itself, and AWS states that on-demand inference on customized Nova models "is charged at the same rate as the non-customized models" — the Fireworks contract shape, not the rental one. The limits are real and narrow: Nova Micro, Lite, 2 Lite and Pro plus Meta Llama 3.3 70B Instruct, in US East (N. Virginia) or US West (Oregon) only, and the model must have been customized on or after 16 July 2025. If your base model is on that list, none of the rental arithmetic above applies to you.
Second, Bedrock Custom Model Import lets you train elsewhere, import Hugging Face-format weights, and invoke them on demand — no Provisioned Throughput. It bills per Custom Model Unit per minute over five-minute windows starting from the first successful inference call, at $0.05718 per CMU-minute in US East and US West, with AWS noting that a Llama 3.1 70B 128K model needs 8 CMUs. That is $27.45 an hour per running copy — but copies scale with demand, so a bursty 8-hours-a-day workload lands near $6,600 a month instead of $28,908, and a genuinely idle one lands near zero.
The training run, meanwhile, barely registers. Together AI charges $0.48 per 1M tokens for LoRA supervised fine-tuning up to 16B, $1.50 for 17–69B and $2.90 for 70–100B, with a $4.00 minimum per job. Eight thousand examples of 800 tokens over three epochs is 19.2M training tokens — $28.80 on a 17–69B model. Databricks Mosaic AI publishes worked examples at the other end of the range: Llama 3.1 70B on a 10-million-word corpus is 225 DBUs, or $146.25, and a 500-million-word run is $7,150. Even the expensive case is under one month of Azure hosting.
Why the Tuned Endpoint Stops Winning at 1,750 Output Tokens
Fine-tuning shortens the prompt, but Vertex's 1.5× multiplier applies to output tokens too — and fine-tuning does not make answers shorter. That produces a hard crossover most business cases miss.
Fine-tuning is cheaper only while the input savings outrun the output markup:
p_in × (I_rag − 1.5 × I_ft) > 0.5 × O × p_out
Working in dollars per million tokens so both sides share a unit: with $1.50 input, $9.00 output, 6,000 RAG input tokens and 500 tuned input tokens, the left side is 1.50 × (6,000 − 750) = 7,875 and the right side is 4.5 × O. They meet at O = 1,750 output tokens.
Below that, the tuned endpoint is cheaper. Above it, you are paying 50% more for every token of a long answer to avoid retrieval you could have run for $0.0063. A summarisation, drafting or report-generation workload with 2,000-token outputs is more expensive fine-tuned on Vertex than it is with RAG, and no amount of prompt shortening rescues it. Run this calculation with your own measured output length before you approve the training job — not with the vendor's.
On Fireworks, where there is no multiplier, the crossover does not exist and the tuned model always wins on tokens. That is a serving contract difference, not a technology difference, and it is worth more than any benchmark in the decision.
Prompt Caching Takes a Third of the Prize for Free
Caching removes the same stable prompt prefix that fine-tuning removes, with no dataset, no training run and no staleness. It is the first thing to try and it is almost always skipped.
In the reference workload, 1,500 of the 6,000 input tokens are a fixed system prompt and few-shot block. Cached at Gemini 3.5 Flash's $0.15 per 1M instead of $1.50, that slice drops from $0.00225 to $0.000225 per query — about $101 a month, or 29% of fine-tuning's entire $354 saving, for the cost of restructuring a prompt. Anthropic's cache read rates are steeper still: $0.10 per 1M against $1.00 input on Haiku 4.5, a 90% discount.
Caching cannot touch retrieved chunks, because they change per query. That is precisely the boundary: the cacheable part of your prompt is the part fine-tuning was going to eliminate anyway.
The Break-Even in Documents Is About 44,000 Tokens
Below roughly 44,000 tokens of corpus, don't build retrieval at all — cache the whole thing.
Gemini charges $0.15 per 1M cached tokens read plus $1.00 per 1M tokens per hour of cache storage. Holding a corpus of C tokens resident costs C × 730 / 1M dollars a month in storage, which across 50,000 queries is C × $0.0146/1M per query, on top of C × $0.15/1M to read it — about C × $0.1646/1M all in. RAG's retrieval slice is 4,200 tokens at $1.50/1M, or $0.0063, plus $0.001 of index amortisation once Pinecone's $50 minimum is spread across 50,000 queries — so $0.0073.
Set them equal: C ≈ 44,000 tokens, or roughly 70 to 75 documents at 600 tokens each. Under that, a cached corpus beats a vector database, a chunker, a reranker and a retrieval failure mode. Over it, retrieval wins and keeps winning, because RAG's cost is flat in corpus size while the cache is linear.
The upper bound on the other side is where fine-tuning stops being an option regardless of price. FineTuneBench (Wu, Wu and Zou, Stanford, November 2024) tested five commercially fine-tunable frontier models including GPT-4o and Gemini 1.5 Pro and found 37% average generalization accuracy on ingesting novel information, falling to 19% when updating knowledge the model already held. Their conclusion is blunt: current commercial fine-tuning services fall short of "reliable knowledge infusion in common scenarios." Ovadia et al. reached the same place — "RAG consistently outperforms it, both for existing knowledge encountered during training and entirely new knowledge."
So the document break-even is not really a curve. It is two cliffs: under ~75 documents you don't need retrieval, and over a few thousand facts you cannot train them in reliably at any price.
Refresh Is Where Fine-Tuning Actually Loses
Re-indexing a 24M-token corpus costs $3.60 and completes in about an hour. Retraining costs a fortnight of somebody's attention.
At Gemini Embedding's $0.15 per 1M tokens, a full re-embed of the reference corpus is under four dollars, and an incremental update of the documents that changed is fractions of a cent. Freshness is measured in minutes. A permissions change or a document retraction propagates immediately, which is the access-control property that decides most regulated RAG builds in the first place.
The fine-tuned equivalent is a new dataset delta, a new training run, a full regression evaluation, a deployment swap and — on Azure or Bedrock — a second hosted deployment running in parallel during cutover, at full hourly rate. The compute is still $29. Everything around it is not.
And the 19% figure above is the reason this is not merely slower but structurally unsound: fine-tuning is worse at correcting what a model already believes than at teaching it something new. A knowledge base is mostly corrections. If your policy changed in July, retraining is the least reliable way to make the model stop citing the June version.
What Evaluation Costs, and Why It Isn't Tokens
Both approaches need the same gold set; only fine-tuning needs it re-run against capabilities you weren't trying to change.
The token cost of evaluation is trivial. A 500-item regression suite generating answers on the reference prompt costs about $6.30, and judging them with a Sonnet 5-class model at $2 in / $10 out adds roughly $3.50 — under $10 a full cycle. JPMorganChase ranked 62 retrieval configurations for about $800 by pooling and reusing judgments, which is the right order of magnitude for a serious retrieval bake-off.
The real asymmetry is structural. With RAG you can swap the chunker, the embedding model or the reranker and re-run the same suite in minutes; each hypothesis is a config change. With fine-tuning each hypothesis is a training run plus a redeployment, and your evaluation has to cover general capability as well as the target task, because a tuning run can quietly degrade behaviour nobody was testing. Budget three to five times the evaluation surface, and put it behind a regression gate you actually enforce.
Now the queries break-even. At $0.0071 saved per query, a cheap dataset — mined from existing tickets, synthetically expanded, 40 hours of review at a fully loaded $120/hour — is $4,850 all in and repays after 683,000 queries, about 14 months at this volume. A carefully human-labelled 8,000-example set at 200 hours is $24,050 and repays after 3.4 million queries, or roughly 68 months. Both assume the corpus never changes, which it will.
Where the Hybrid Genuinely Wins
Fine-tune the behaviour; retrieve the facts. This is not a compromise position — it is what the last three years of evidence actually says, and the research converged on it from three directions.
Yang et al. (January 2026) tested multi-hop question answering over novel knowledge and found unsupervised continual pretraining gave "only limited gains," RAG gave "substantial and consistent improvements," and supervised fine-tuning "achieves the highest overall accuracy across models and datasets" — the split being that SFT teaches the reasoning pattern while retrieval supplies the facts. Ibrahim, Rozanov and Rei, under review at ICLR 2026, distilled retrieval hints into model weights and reported ALFWorld success rising from 79% to up to 91% and WebShop scores from 61 to 72, while using 10–60% fewer tokens than the retrieval-augmented teachers they learned from. That token reduction is the serving-cost argument for fine-tuning, quantified honestly.
And on cost specifically, Sturm et al. (May 2026) evaluated both on two closed automotive datasets under an extended Cost-of-Pass framework and concluded: "RAG emerges as the most effective and cost-efficient adaptation method for both closed- and open-source models," with open-weight models reaching premium-model quality once retrieval was added.
The practical reading: fine-tune a small open-weight model on Fireworks or Together to enforce output schema, tool-call discipline, refusal behaviour and how to handle a retrieval miss — then feed it retrieved context at inference. You get the shorter prompt and the fresh facts, and on a no-premium serving contract you pay for neither twice. That is the only version of "hybrid" worth the operational overhead.
Who Should Not Pick Each Option
- Don't pick RAG if your problem is format, not facts — if the model knows the answer and gets the JSON schema wrong, retrieval adds 4,200 tokens a query and fixes nothing. Also skip it under ~75 documents; cache them instead.
- Don't pick a Vertex tuned endpoint if your average output runs past ~1,750 tokens. The 1.5× multiplier on output will eat the input savings and then some.
- Don't pick Azure OpenAI fine-tuning for a workload under a few hundred thousand queries a month. $1,241 of hosting against a $354 saving is a losing trade at this volume, and the 15-day idle deletion punishes exactly the intermittent workloads that would otherwise tolerate it.
- Don't pick Bedrock Provisioned Throughput unless you are running sustained, high, predictable throughput — a one- or six-month term cannot be deleted before it expires, and a model unit has a fixed tokens-per-minute ceiling, so you cannot amortise it indefinitely. Check first whether your base model qualifies for on-demand custom deployment (Nova or Llama 3.3 70B, Virginia or Oregon); fall back to Custom Model Import if it does not. Provisioned Throughput should be the last option you reach for on AWS, not the first.
- Don't pick OpenAI fine-tuning at all if you are a new customer. That door shut on 7 May 2026, and the last one closes 6 January 2027.
- Don't pick fine-tuning of any flavour to store facts that change. 19% generalization on knowledge updates is not a number you build a compliance answer on.
The loser is Bedrock's Provisioned Throughput path, and it loses on structure rather than on price. Putting a custom model on committed capacity converts a variable cost into a fixed rental with an exit penalty, at 23× to 43× the RAG bill for the reference workload, for a customization technique the research says is the wrong tool for knowledge anyway. It is no longer the only way to serve a Bedrock fine-tune — on-demand custom deployment and Custom Model Import both bill by use, and either is the better default. What actually deserves the warning here is the documentation: AWS's Provisioned Throughput page still tells you a customized model requires it, and a buyer who takes that sentence at face value will rent capacity they never needed.
What to Do in the Next 90 Days
This Week:
- Pull your actual average output token count from your gateway logs and run the crossover formula above. If it is over 1,750 tokens, the fine-tuning business case on Vertex is already dead and you can stop.
- Grep your prompt for the stable prefix — system instructions, few-shot examples, tool definitions — and put it behind prompt caching. That is roughly a third of fine-tuning's benefit for an afternoon of work.
- Audit for orphaned fine-tuned deployments. On Azure they bill hourly at zero traffic; on Bedrock they bill hourly under commitment. Somebody's 2025 experiment is very likely still running.
This Month: 4. If you are an existing OpenAI fine-tuning customer, decide before 6 January 2027 whether you are migrating to Vertex, an open-weight path, or back to retrieval. After that date you cannot create new jobs, and your existing models live only as long as their base models do. 5. Build the 500-item gold set. It is the expensive artefact in both approaches and the only one that transfers between them.
Before Q1 Close: 6. If you still want a custom model, run the bake-off on Fireworks or Together, where LoRA training is $0.48–$10 per 1M tokens and serving carries no premium. Prove the accuracy delta against a RAG baseline on the same gold set before you commit to any hourly contract. 7. Price the refresh, not the launch. Ask what it costs to make the model stop saying the June answer, and how many times a year that happens.
The Bottom Line
Every previous platform cycle taught the same lesson and enterprises relearned it each time: the acquisition price is not the price. Nobody was ruined by the licence fee for a data warehouse; they were ruined by five years of ETL. Fine-tuning is that shape exactly. A $28.80 training run looks like a decision you can make in a stand-up, and it commits you to a serving contract, a retraining cadence, an expanded regression suite and a deployment you have to remember to turn off.
Retrieval keeps the facts outside the weights, where they can be corrected in an hour for less than four dollars. Fine-tuning moves them inside, where correcting them is a project. That is not an argument about accuracy. It is an argument about which mistakes are cheap to fix.
Price the contract, not the GPU.
Continue Reading
- What RAG Actually Costs: $1,308 a Month at 10M Tokens/Day
- RAG Build vs Buy: Buy the Index. Build the Eval Set.
- Vector Database Pricing: Only pgvector Publishes a Rate
- Pinecone vs Weaviate vs pgvector: Stay on Postgres
- Braintrust vs Langfuse vs Promptfoo: Don't Pay for the Gate
- vLLM vs TensorRT-LLM vs SGLang: Default to vLLM
- GPU Hour Pricing: A $3.44 Rate Cost $68.80 to Use
- JPMorgan Ranked 62 Retrievers for $800. Pool the Judgments.