If your self-hosting business case divides the GPU's hourly rate by its benchmark throughput, it assumes the card is busy every hour of the month, and internal traffic that stops at 6pm never gets close. Run gpt-oss-120b on rented H100s for a workforce that logs off in the evening and you pay roughly twice what Together AI or Amazon Bedrock charge for the same model. A B200 can beat the API on paper, but only if you fill it during the day and keep no spare.
The verdict: use the API for gpt-oss-120b and DeepSeek V4 Pro unless you have a residency, air-gap or control requirement, or a batch workload that keeps the cards busy overnight. If you self-host for one of those reasons, budget for the premium and don't present it as a saving.
All prices below were checked on each vendor's live pricing page on October 23, 2026. The traffic shape throughout is the one SemiAnalysis benchmarks: single-turn chat, 8,000 input tokens and 1,000 output tokens per request. At Together's list price that blends to $0.20 per million total tokens.
| Option (checked Oct 23, 2026) | Listed price | What it takes on 8k-in / 1k-out traffic | Verdict | Who should not pick it |
|---|---|---|---|---|
| Together AI serverless, gpt-oss-120b | $0.15 in / $0.60 out per 1M tokens | $0.20 per 1M total tokens, no floor | Default | Teams whose data cannot leave their own network |
| Amazon Bedrock, gpt-oss-120b | $0.15 in / $0.60 out per 1M tokens | $0.20 per 1M total tokens | Default on AWS | Anyone who needs to pin the serving stack |
| Lambda 1x H100 SXM, self-hosted | $4.29 per GPU-hour | 5,958 tokens/s, every hour of the month | Loser | Anyone with business-hours traffic |
| RunPod Secure Cloud H100 SXM, self-hosted | $3.99 per GPU-hour | 5,542 tokens/s, every hour | Loses on 9-to-5 load | Anyone without overnight batch work |
| Lambda 1x B200, self-hosted | $6.99 per GPU-hour | 9,708 tokens/s, every hour | Wins only when saturated by day with no spare | Anyone with an availability SLA |
| DeepSeek API, V4 Pro | $1.32 in / $3.96 out per 1M (peak); half that off-peak | $1.61 per 1M total tokens at peak | Default for V4 Pro | Teams barred from a China-hosted API |
| Self-hosted V4 Pro on 8x B200 (Lambda) | $53.52 per node-hour | About $39,070 a month per replica | Residency or air-gap only | Anyone spending under $39,000 a month on the API |
What Is the Break-Even Formula for Self-Hosting an LLM?
The break-even throughput is the GPU's hourly cost divided by the API's price per million tokens, times 1,000,000, divided by 3,600 seconds. It gives the tokens per second a GPU must sustain, averaged over every hour you pay for it, to match the API. Below that rate the API is cheaper.
Worked for gpt-oss-120b on one H100 SXM at Lambda's $4.29 per GPU-hour:
- Against the output price alone ($0.60 per million at Together): $4.29 ÷ $0.60 = 7.15 million tokens an hour, or 1,986 output tokens per second.
- Against the blended price for 8k-in, 1k-out traffic ($0.20 per million total tokens): $4.29 ÷ $0.20 = 21.45 million tokens an hour, or 5,958 total tokens per second.
Use the blended figure. Your benchmark will almost certainly report total tokens, and input tokens dominate most enterprise traffic. OpenRouter's listing shows the Bedrock route at the same $0.15 and $0.60, so the comparison holds whether you buy from Together or AWS.
The same arithmetic for the other rates on the table, at $0.20 blended:
| GPU and rate (Oct 23, 2026) | Break-even, total tokens/s per GPU |
|---|---|
| RunPod Community Cloud H100 SXM, $2.69/hr | 3,736 |
| RunPod Secure Cloud H100 SXM, $3.99/hr | 5,542 |
| Lambda 1x H100 SXM, $4.29/hr | 5,958 |
| Together dedicated H100, $5.49/hr | 7,625 |
| Lambda 1x B200, $6.99/hr | 9,708 |
| Fireworks on-demand H100, $8.00/hr | 11,111 |
The RunPod rates are from its pricing page. The dedicated rates are Together's $5.49 per H100-hour and Fireworks' $8.00 per H100-hour. Renting a dedicated endpoint from an inference provider to save on tokens is the worst version of this trade: you take on the utilisation risk and still pay a managed-service margin on the GPU. Fireworks and Together sell dedicated capacity for latency isolation and custom weights, and neither rate makes sense as a way to cut cost.
How Many Tokens per Second Does gpt-oss-120b Actually Produce?
On an H100, gpt-oss-120b produces between about 2,500 and 9,000 total tokens per second per GPU, and where it lands depends entirely on how fast each user's response has to stream. That trade is the number most business cases leave out. A GPU serving more users at once produces more tokens in total and fewer per user.
SemiAnalysis's InferenceX run for gpt-oss-120b on H100 (vLLM, FP4, 8k/1k chat, runs dated March 27 to May 17, 2026) reports:
| Speed per user (tokens/s/user) | Throughput (tokens/s/GPU) | Share of the month a $4.29/hr H100 must run at that rate to break even |
|---|---|---|
| 50 | 8,973 | 66% |
| 100 | 5,863 | 102% (cannot break even) |
| 150 | 3,931 | Cannot break even |
| 200 | 2,558 | Cannot break even |
At 100 tokens per second per user, roughly the speed of a responsive chat product, an H100 at Lambda's list price cannot match the API even running flat out all month. At 50 tokens per second per user it needs two thirds of every hour of the month at full batch.
The B200 numbers are several times higher. The InferenceX B200 vs H100 comparison on the same 8k/1k FP4 workload shows 36,090 tokens/s per B200 at 90 tokens/s/user against 6,362 on the H100, and 17,465 against 2,636 at 197 tokens/s/user. The InferenceX ranking page lists the B200 at 44,489 tokens/s per GPU at 50 tokens/s/user. Against the $9,708 break-even at Lambda's $6.99, a B200 needs 22% of the month at 50 tokens/s/user, 27% at 90 and 56% at 197.
Three caveats before you take those numbers into a budget:
- InferenceX costs use SemiAnalysis TCO rates. The H100 page prices the GPU at $1.17 an hour ("Owning at Large Hyperscaler Volume") or $2.00 retail. Lambda lists $4.29. Recompute the per-million figure with your own rate before quoting it.
- InferenceX reports the best of many tuned configurations. The H100 page benchmarked 43 of them. In the GPUStack benchmark that Cohorte cites, default vLLM settings for gpt-oss-120b produced 2,901 output tokens per second against 7,648 tuned, a 2.6x gap. Your first month will look like the default.
- The throughput counts total tokens. The ranking page says its cost is "$ per million total (input plus output) tokens." Compare it to a blended API price and never to the output price alone.
Why Does Utilisation Decide the Whole Answer?
Utilisation decides it because you pay the GPU for every hour you hold it and the API for every token you send. A card that is idle from 6pm to 8am and all weekend still bills, and those idle hours are where the API's advantage comes from.
The published example is Cohorte's September 20, 2026 analysis. Two H100 SXMs on Lambda at the $4.19 per GPU-hour then listed came to $6,117 a month. At 7,648 output tokens per second they produce output at about $0.30 per million when run flat out around the clock, half of Together's $0.60. Break-even needs full capacity for 50.7% of all hours. Run at full capacity through working hours only and the cost is $1.28 per million output tokens, which the author calls "twice what the provider charges."
The same thing shows up in total-token terms. Define business hours as 10 hours on 22 weekdays: 220 of the month's 730 hours, or 30%. Rent a Lambda H100 around the clock, saturate it at 50 tokens/s/user every business hour, and leave it idle otherwise:
- Monthly cost: $4.29 × 730 = $3,132
- Monthly tokens: 8,973 × 3,600 × 220 = 7.1 billion
- Cost: $0.44 per million total tokens, 2.2x the API's $0.20
That is the best case. It assumes the card is saturated for every one of those 220 hours. Traffic you provision for at peak averages lower, and every point it drops below saturation raises the 2.2x.
Run the same day on a single B200 and it comes out at $0.145 per million at 50 tokens/s/user, below the API. Add one spare for availability and the pair costs $0.29, 1.45x the API. At 90 tokens/s/user the spare takes it to $0.36, 1.8x.
The strongest counterargument is scale-to-zero. Lambda's on-demand instances carry no commitment, so you could shut the cards down at night and pay for 220 hours. A single H100 then needs to average 66% of its saturated throughput across the working day to match $0.20. Two B200s (one spare) need traffic averaging 44% of one card's saturated throughput at 50 tokens/s/user. A heavy, steady internal workload can reach that. It also assumes on-demand capacity is available every morning when you ask for it, which the on-demand tier does not promise, and the Bedrock vs EKS break-even analysis published October 23, 2026 notes that "keeping one node warm puts you back on the 24-hour cost." That piece puts break-even for gpt-oss-120b on one AWS H100 at about 7,300 tokens per second sustained around the clock and concludes: "Unless your traffic is that high and that steady, Bedrock costs less."
Overnight batch work changes the answer. If you have batch work that can soak up the idle hours (embedding a document corpus, running evals, overnight extraction jobs), the denominator in every calculation above grows and the 30% ceiling disappears. Without that batch work, the business-hours ceiling holds.
What Changes for a Model That Needs a Whole Node?
A model that needs a whole node raises the floor from one GPU to eight, and the break-even moves from a throughput question to a spend question. DeepSeek V4 Pro shows it clearly.
The DeepSeek V4 Pro model card lists 1.6 trillion total parameters, 49 billion active, FP4 experts with FP8 elsewhere, under the MIT license. NVIDIA's Dynamo recipe deploys it on 8x B200 (full 1,048,576-token context) or 8x H200, where HBM caps the context at 86,016 tokens. The checkpoint is about 865 GB, and the recipe allows up to about 90 minutes for first launch to load weights across the eight GPUs and warm up. A cold start that long rules out shutting the node down overnight.
Compare that with gpt-oss-120b, which the model card says runs "on a single 80GB GPU." With 117 billion total parameters and 5.1 billion active, the hardware floor is one card, so the utilisation question is about one card's hours. (For why total parameters set the memory bill on a mixture-of-experts model while active parameters set the speed, see our MoE sizing piece.)
The node-scale arithmetic, at Lambda's 8x B200 rate of $6.69 per GPU-hour:
- One replica: $53.52 an hour, $39,070 a month. Two for redundancy: $78,140.
- DeepSeek's API charges $1.32 per million input tokens on a cache miss and $3.96 per million output at peak, half that off-peak. Peak is only 01:00 to 04:00 and 06:00 to 10:00 UTC on weekdays, so US business-hours traffic lands almost entirely in the off-peak window. On 8k/1k traffic that blends to $1.61 per million total tokens at peak and $0.81 off-peak.
- To match one unreplicated node you need 24 billion tokens a month at peak rates, or 48 billion if your traffic falls in DeepSeek's off-peak window, as a US daytime workload mostly does.
Unless your DeepSeek bill already runs near $39,000 a month, a self-hosted V4 Pro replica costs more than the API before you pay anyone to run it. And DeepSeek bills cache hits at $0.044 per million at peak, so agent traffic with long, stable prefixes makes the API cheaper still.
The node can produce the tokens. 24 billion a month is about 1,150 tokens per second per GPU, and InferenceX's B200 run on its AgentX agentic-coding workload reports far more than that per GPU. That workload replays agent sessions with their KV-cache prefix reuse intact, so its per-million cost is not comparable to DeepSeek's cache-miss price. If you need V4 Pro off DeepSeek's own servers, Together lists it at the same $1.32 and $3.96, with no node to rent.
Which Costs Does the GPU Hourly Rate Leave Out?
The hourly rate leaves out the people, the upgrades, the evaluation and the spare, and on a small deployment those can cost more than the GPUs.
Start with people. The Bureau of Labor Statistics puts the median software developer wage at $135,980 (May 2025), before benefits. A quarter of one engineer's time on the serving stack is about $2,830 a month in wages alone. A 24x7 on-call rotation takes several people, not a fraction of one.
The serving stack moves fast. The vLLM releases page lists v0.30.0 on September 22, v0.31.0 on October 5 and v0.32.0 on October 18, each with breaking changes: removed flags, a renamed quantization shorthand, dropped Python 3.10 support, changed default CORS origins. Three releases in four weeks means somebody reads release notes and retests at least monthly or falls behind on fixes. (Our runtime comparison covers why vLLM is still the default to carry.)
Every stack upgrade, quantization change or new checkpoint needs your eval suite run again before it takes traffic. The API side has the same problem in a different form: DeepSeek's pricing page notes that the deepseek-v4-flash name is now served by DeepSeek-V4.1-Flash-1017. A hosted model can change under a stable name, which is the failure we covered when it happened before.
An availability SLA needs N+1. On a model that fits one GPU, N+1 doubles the GPU bill and halves the utilisation of each card, which is why the single-B200 win above disappears.
Add it up for the smallest serious gpt-oss-120b deployment: two Lambda H100 SXMs at $4.29 (one spare) is $6,263 a month, plus a quarter engineer at $2,830, about $9,090. At the API's $0.20 blended rate that money buys 45 billion tokens a month, equal to about 17,300 tokens per second sustained around the clock. Your traffic has to be that large before self-hosting is even level.
When Should You Self-Host Anyway?
Self-host when a requirement forces it: data residency, an air-gapped network, a model you have fine-tuned that no provider serves, or a need to pin the exact weights and serving stack for audit. In those cases you pay a premium for control, and the business case should say how much.
- Air-gap and classified networks. No API reaches them. The stack choice there is about licences and support, which our air-gapped stack guide covers.
- Residency rules that rule out every hosted region. Check first. gpt-oss-120b and V4 Pro are sold by several hosts, and our provider comparison lists their data-retention terms.
- Version control. If a model update reaching you unannounced is an audit finding, owning the weights fixes that.
- GPUs you already own. Hardware bought for training and sitting idle has a marginal cost close to its power bill. The formula still applies, with your real marginal rate in place of a rental price.
Write the business case as "we pay X% over the API for residency." A case that claims savings on a daytime workload will miss its own forecast within a quarter.
How Do You Decide Without Regretting It?
The criteria that predict regret are the daytime duty cycle, the availability requirement and the model's hardware floor. Price per GPU-hour matters less than any of them.
- Measure the duty cycle first. Pull a month of request logs and compute tokens per hour across all 730 hours. If more than 70% of the month is near-idle, the H100 case is closed.
- Fix the per-user speed. Decide whether users need 50 or 100 tokens per second per user, then read throughput at that point and no other. At 100, an H100 at list price never breaks even against $0.20.
- Decide on the spare before you price. If an outage needs a fallback, either price N+1 or route failover to the API, which an LLM gateway can do.
- Price the node, not the card, for big MoE models. For V4 Pro the question is whether your API bill exceeds $39,000 a month. Throughput is secondary.
- Find the night shift. Batch work that can run from 6pm to 8am is what turns a losing daytime case into a winning one.
What changes the answer: a B200 rate well below list, a committed rental you have already signed (see our reserved vs spot analysis), a steady 24-hour workload, or an API price rise on the model you depend on.
This Week:
- Export one month of token counts by hour and calculate the share of hours with real load.
- Recompute any InferenceX or vendor $/M figure with your actual GPU rate and the blended API price for your input/output mix.
This Month:
- If self-hosting is still on the table, benchmark default vLLM on your own prompts at your target per-user speed, and use that number instead of a tuned leaderboard result.
- List the batch jobs that could run overnight and estimate their token volume. Without them, keep the workload on the API.
Before the Next Budget Cycle:
- If residency or air-gap forces self-hosting, write the premium into the budget as a line item for compliance.
The Bottom Line
Data centre teams ran the same arithmetic when cloud instances arrived: owned servers looked cheap until someone counted the idle hours and the staff. Token APIs for open models are now priced near the cost of a saturated GPU, and a business-hours workload saturates nothing for 70% of the month. Use the API for gpt-oss-120b and DeepSeek V4 Pro by default. Self-host when a requirement forces it or a night shift of batch work fills the cards, and price it as a premium.
Run the 730-hour log before you sign a GPU rental.
Continue Reading
- Reserved vs Spot GPUs: Commit Six Months, Not Twelve
- Provisioned Throughput Pricing: Monthly Terms Need 99% Utilization
- Together vs Fireworks vs Groq: Price the Tool Call, Not the Token
- Inference Cost per Million Tokens: Price the Task, Not the Rate
- FP8 vs NVFP4 vs INT4-AWQ: AWQ Outran NVFP4 on 3 of 4 Models
- Self-Host the Vector DB for Residency. Not for the Bill.