The enterprise AI graveyard is filling up — and the cause of death isn't hallucinations, governance failures, or integration complexity. It's the monthly API bill.
A VentureBeat Pulse survey found that 29% of enterprise AI projects are killed not by model failure, but by token costs. Think about that number for a moment. Nearly one in three AI initiatives — projects that often had executive sponsorship, technical talent, and real business cases — die because the math on inference spend didn't work out.
Meanwhile, companies plan to spend 1.7% of revenue on AI in 2026, and 80-85% of enterprises miss their AI infrastructure cost forecasts by more than 25%, according to The State of AI Cost Governance report. The gap between what leaders expect to spend and what they actually spend has become the defining challenge of enterprise AI at scale.
Shopify has built a systematic answer to this problem. And it's worth studying.
The Token Cost Death Spiral
Here's how it typically unfolds at large enterprises.
A team identifies a high-value use case — customer support automation, sales proposal generation, contract review. They prototype with a frontier model (Opus 4, GPT-5+, Gemini Ultra) because those models deliver the best results. The prototype works. Executives love the demo. The project gets greenlit.
Then it hits production scale.
What cost $500/month in testing costs $50,000/month at enterprise volume. The model that seemed like a reasonable expense at 10,000 API calls per day becomes unsustainable at 1,000,000. The CFO asks for the ROI calculation. The numbers don't close. The project gets deprioritized or killed outright.
The instinct at this point is usually wrong: teams either try to engineer their way out with prompt compression and caching, or they downgrade to a smaller model and watch quality fall off a cliff. Neither solves the actual problem.
Shopify's engineering leadership has articulated the real issue clearly. The mistake isn't using frontier models. It's using frontier models for everything — including the narrow, repetitive subtasks where a smaller specialized model would outperform them on accuracy, cost, and latency simultaneously.
What Shopify Built: The Universal Distillation Platform
Farhan Thawar, VP and Head of Engineering at Shopify, describes a two-layer approach that separates exploration from production.
The first layer is the frontier model. Opus 4, GPT-5+, and equivalents remain the right tool for complex reasoning, novel problem-solving, and tasks where the surface area of possible outputs is genuinely wide. When Shopify's engineering teams are building something new, evaluating unfamiliar problem spaces, or handling edge cases that require broad world knowledge, they reach for the most capable model available.
The second layer is the Universal Distillation Platform (UDP). Once a use case is well-defined — once the team understands exactly what the model needs to do for a specific narrow task — they use the frontier model to generate training data and distill that intelligence into a fine-tuned open-source model. The current go-to is Qwen, though the platform is model-agnostic.
The results are striking: 2x to 30x cheaper, faster, and in many cases more accurate than calling the frontier API for that same narrow task. Shopify currently runs half a dozen of these distilled models in production, with more being added regularly. The distillation process takes roughly a day, with evaluations baked into the pipeline.
For CIOs and CTOs considering this approach: the insight is that the frontier model's job is not to be your production inference engine for every task. Its job is to be the teacher. The smaller model is the student that graduates to production.
The Business Case for Distillation (What CFOs Need to Hear)
Let's do the math that enterprise finance teams actually care about.
Assume a mid-size enterprise runs an AI-powered customer support function at 500,000 queries per month. At frontier model pricing (~$15 per million output tokens, with average responses of ~500 tokens), that's roughly $3,750/month just on inference for responses — before you count input tokens, orchestration, and infrastructure overhead.
After distillation to a fine-tuned smaller model running on your own infrastructure or a cheaper inference tier, the cost-per-call drops by 10-30x. The same workload now costs $125-375/month in inference. Annualized, that's a swing from $45,000 to roughly $1,500-4,500 in model inference costs for that use case alone.
Multiply that across five or ten production AI use cases at enterprise scale, and you're looking at potential seven-figure annual savings. Not hypothetically — Shopify is realizing these numbers in production today.
Thawar has been direct about the incentive structure: "29% of companies say it's token costs that are killing projects, not model failure." The implication is that enterprises don't have a model quality problem — they have a cost architecture problem. And the fix is structural, not incremental.
The LLM Proxy: The Other Half of Shopify's Infrastructure
The Universal Distillation Platform doesn't operate in isolation. Shopify pairs it with an internal LLM proxy that routes token traffic across every major provider with automatic failover.
This matters for enterprise AI leadership for two reasons.
First, it eliminates single-provider dependency. When any one LLM provider goes down — and they do — traffic automatically reroutes. For business-critical applications, this isn't optional. Enterprises that have hardwired their AI applications to a single provider are one outage away from discovering that dependency.
Second, it creates cost arbitrage opportunities. Different providers price differently for different tasks and token volumes. A centralized proxy lets engineering teams optimize routing based on cost, latency, and capability requirements without rewriting application code. When a new model drops at a lower price point, the proxy enables rapid migration without touching application logic.
For CTOs: this is an infrastructure decision that pays compounding dividends. The proxy is the foundation for model diversity; the UDP runs on top of it. Together, they give Shopify the ability to adopt any frontier model as a teacher and deploy any specialized model in production — without being locked into any single provider's pricing or availability.
River: The Third Piece — Agentic Infrastructure That Improves Itself
Beyond distillation and the proxy, Shopify has deployed something they call River: an internal agentic system that operates through Slack.
River queries Shopify's data warehouse, reads their project management systems, and answers questions from across the engineering and product organization. What makes it notable is the feedback loop: when an engineer corrects one of River's answers, River updates its own behavior. It's a production AI system that improves from operational use, not just from scheduled retraining.
For enterprise leaders thinking about internal AI deployment, River represents a model worth examining. The tendency in enterprise AI is to build AI tools for customers while treating internal operations as a secondary use case. Shopify has inverted this. Their most sophisticated AI infrastructure — the proxy, the UDP, River — is built for and deployed by the people who build Shopify. That internal usage generates the feedback, evals, and optimization pressure that improves the system.
The lesson: enterprises that build AI for customers first and internal users second miss the fastest feedback loop available to them.
What the McKinsey Data Says About Where Most Enterprises Stand
Shopify's approach looks sophisticated relative to where most enterprises actually are.
McKinsey's State of AI 2026 report found that 78% of organizations use AI in at least one business function — up from 55% just two years ago. That adoption rate is genuinely impressive.
But the same report flags a stark reality: fewer than one in ten enterprises can point to a GenAI deployment that is delivering measurable, sustained business value at scale.
The gap between "using AI" and "extracting value from AI at scale" is exactly the gap that token cost architecture determines. Organizations that are evaluating frontier models in pilots will get impressive demos. Organizations that build the infrastructure Shopify has built — the proxy, the distillation pipeline, the feedback loops — will be the ones that close the ROI gap.
The enterprises that can't show CFOs a credible path from AI investment to measurable return by the end of 2026 will face budget scrutiny in 2027 planning cycles. The ones who have cracked cost architecture won't.
Three Decisions Enterprise Leaders Should Make Now
1. Audit your current AI inference spend by task type.
Before you can distill, you need to know what you're actually using frontier models for. Most enterprises don't have this visibility. Run a 30-day audit of your AI API calls segmented by use case. You'll almost certainly find that 70-80% of your spend is on repetitive, narrow tasks that are candidates for distillation.
2. Establish model routing infrastructure before you need it.
The LLM proxy is not an optimization for later — it's a prerequisite for model strategy. If you don't have centralized routing today, every new model you adopt creates a new integration dependency. Build the proxy now while your model footprint is still manageable.
3. Define your distillation pipeline criteria.
Not every use case is ready for distillation. Good candidates share three characteristics: they're high volume, well-defined in scope, and evaluable with automated metrics. Shopify's UDP took roughly a day to distill a frontier model into a production-ready specialist. Your timeline will vary based on your ML infrastructure maturity — but you need criteria before you can identify candidates.
The Strategic Frame
There's a useful mental model for how frontier models and distilled specialist models relate to each other in a mature enterprise AI stack.
Think of frontier models as research and development infrastructure. You use them to explore, prototype, and generate training signal. They are expensive by design — they're the output of billions of dollars in compute and data. You don't run your production workloads on R&D infrastructure.
Distilled specialist models are production infrastructure. They're optimized for a specific task, cheap to run, fast to respond, and tuned to your data. They don't need to know everything — they need to be very good at one thing.
The mistake most enterprises are making right now is treating R&D infrastructure as production infrastructure because the demo worked. Shopify's engineering team figured out that the demo is where you validate the use case. The UDP is where you build the production system.
The Bottom Line
Shopify's Universal Distillation Platform isn't an exotic innovation available only to trillion-dollar commerce platforms. The underlying techniques — distillation, fine-tuning, LLM proxy routing — are available to any enterprise with a machine learning team and the architectural discipline to apply them.
What Shopify demonstrates is that the question "which AI model should we use?" is the wrong question. The right question is "what infrastructure do we need to use the right model for each task at the right cost?"
For 29% of enterprise AI projects, the answer to that question came too late. The token bill arrived before the architecture did, and the business case collapsed.
The enterprises that will still be running AI at scale in 2027 are the ones building that architecture now.
Rajesh Beri covers enterprise AI strategy and implementation for technical and business leaders. He writes THE DAILY BRIEF twice weekly.
Follow on Twitter/X or connect on LinkedIn for daily enterprise AI insights.
