By Rajesh Beri | July 12, 2026
For three years, the AI industry poured hundreds of billions into a single obsession: training bigger models. Frontier labs burned through GPU clusters measured in six-figure node counts, raced to trillion-parameter architectures, and consumed more electricity than small countries. The implicit assumption was that whoever trained the best model would capture the most value.
That assumption just got a $1 billion counterargument.
On July 8, 2026, SambaNova Systems announced the first close of a $1 billion Series F round, valuing the Palo Alto-based AI chip maker at $11 billion post-money. The round was led by General Atlantic, with significant investment from Seligman Ventures, T. Rowe Price, and Capital Group. Intel, BlackRock, Qatar Investment Authority, and Vista Equity Partners also participated.
This would be notable on its own. But two details make it structurally significant for every enterprise AI leader.
First, SambaNova raised a $350 million Series E just five months ago. A company doesn't nearly triple its valuation in 150 days unless the market it serves is moving faster than anyone projected.
Second — and this matters more — JPMorgan Chase simultaneously announced it has selected SambaNova as an "inference infrastructure partner," deploying SN40L and SN50 systems for secure, on-premises AI inference. When the largest bank in the United States decides its most sensitive AI workloads belong on dedicated inference hardware inside its own walls — not on NVIDIA GPUs rented through a hyperscaler — the strategic calculus for every regulated enterprise shifts.
"At JPMorganChase, AI infrastructure has to meet a very high bar for performance, control, and reliability," said Darrin Alves, CIO of Infrastructure Platforms at JPMorgan Chase. "We're excited to deploy SambaNova's RDU architecture and looking forward to testing its speed and security for on-prem inference in our demanding enterprise AI workloads."
This isn't a pilot. It's a procurement decision by a $4 trillion bank. And it crystallizes a structural shift that's been building for 18 months: the AI value chain is migrating from training to inference, and the companies that own the inference layer will capture the next decade of enterprise spending.
The Inference Inversion: Why 80% of Your AI Budget Is About to Shift
Here's the number that should reshape your infrastructure roadmap: inference now accounts for 55% of all AI-optimized cloud infrastructure spending, according to Gartner, surpassing training for the first time in early 2026. By 2029, that figure reaches 65%.
But cloud spending tells only half the story. Over a model's production lifecycle, inference represents 80-90% of total compute costs. Training is a one-time event; inference runs every hour of every day for years. Every API call, every agent action, every automated decision — that's inference. As agentic AI workflows proliferate, each task triggers multiple inference calls in sequence, compounding the cost.
The math is brutal and simple. A model that costs $100 million to train might cost $500 million to serve over its production lifetime. And as enterprises move from single-model chatbots to multi-agent systems orchestrating dozens of models simultaneously, inference costs don't scale linearly — they multiply.
This is why the AI infrastructure market is undergoing what I'd call an inference inversion: the center of gravity in AI spending is flipping from "build the model" to "run the model." The global AI inference market hit $117.8 billion in 2026 and is projected to reach $312 billion by 2034 — growing at nearly 13% CAGR even as training costs plateau.
Three forces are accelerating this shift:
1. Agentic AI multiplies inference demand. A traditional chatbot makes one inference call per user query. An agentic workflow — planning, tool-calling, self-correcting — can make 10-50 inference calls per task. As Gartner predicts 70% of enterprises will deploy agentic AI for IT operations by 2029, inference volume will explode exponentially.
2. Model proliferation means multi-model inference. Enterprises aren't running one model anymore. They're running dozens — specialized models for different domains, different languages, different compliance requirements. The infrastructure that can switch between models in microseconds wins. The infrastructure that requires minutes to swap models bleeds money.
3. Regulation demands on-prem control. The EU AI Act's high-risk obligations become enforceable in August 2026. Auditable, transparent, sovereign inference isn't optional for regulated industries — it's a legal requirement. Cloud-based inference through third-party endpoints creates compliance surface area that on-premise deployment eliminates entirely.
The Inference Chip Wars: Who's Winning and Why It Matters
SambaNova's raise didn't happen in a vacuum. The inference hardware market has seen more capital, more exits, and more competitive reshuffling in the past 12 months than in the previous five years combined.
NVIDIA acquired Groq for $20 billion in December 2025, absorbing the LPU (Language Processing Unit) architecture and its deterministic latency capabilities into the Blackwell ecosystem. The combined Blackwell GPU + Groq LPX architecture, announced in March 2026, is NVIDIA's explicit response to the inference-first challengers.
Cerebras went public on May 14, 2026, closing day one at a $56 billion fully diluted valuation — the largest U.S. tech IPO since Snowflake. Its wafer-scale WSE-3 chip, a single 5nm die carrying 44GB of on-package memory, is designed to run massive models on a single piece of silicon. Cerebras is sold out into 2027.
OpenAI and Broadcom unveiled Jalapeño in June 2026, a custom inference ASIC that cuts token costs 50% vs. NVIDIA GPUs. Built in nine months with AI-assisted design, it signals that even the model makers are now vertically integrating into inference hardware.
Qualcomm acquired Modular for $3.9 billion in June 2026, buying LLVM creator Chris Lattner's AI infrastructure startup to break NVIDIA's CUDA software lock on enterprise inference.
And now SambaNova has $1.35 billion in fresh capital ($350M Series E + $1B Series F), an $11 billion valuation, an Intel manufacturing partnership, and JPMorgan Chase as a marquee customer.
The pattern is unmistakable: the inference layer is fragmenting away from NVIDIA's GPU monopoly, and investors are placing massive bets on purpose-built alternatives. The question for enterprise leaders isn't whether to diversify — it's which architecture matches their workload profile.
What Makes SambaNova Different: The RDU Architecture
SambaNova's core differentiation is architectural. Its Reconfigurable Dataflow Unit (RDU) is not a GPU. It's not an ASIC. It's a purpose-built inference processor that moves data through compute units in a pipeline, rather than the instruction-set approach used by CPUs and GPUs.
The practical implications for enterprise deployment:
Speed at low batch sizes. SambaNova claims its SN40L is 4x faster and 2.5x more energy-efficient than an NVIDIA H200 at higher batch sizes, and up to 9x faster at low batch sizes. The newer SN50, shipping in H2 2026, claims 5x the maximum speed and 3x the throughput compared to Blackwell B200 for agentic inference.
Three-tiered memory architecture. The RDU combines SRAM, HBM, and DDR DRAM to support models up to 5 trillion parameters on a single system node — without sharding across dozens of GPUs. This matters enormously for regulated enterprises that need full model containment within their own infrastructure.
Microsecond model switching. Traditional GPU setups take minutes to swap between models. SambaNova's architecture switches in microseconds. For multi-model agentic workflows — where a single request might chain a reasoning model, a code model, and a domain-specific model — this eliminates the latency tax that makes multi-model pipelines impractical on GPU clusters.
Air-cooled, 20 kW per rack. The SN50 SambaRack averages 20 kW, compared to the 40-70 kW typically required by high-end GPU racks with liquid cooling. For enterprises that can't retrofit their data centers with liquid cooling infrastructure, this is a deployment prerequisite, not a nice-to-have.
Full-stack, no CUDA required. SambaNova ships hardware, software (SambaNova Suite), and pre-integrated models as a complete platform. It uses standard open interfaces like vLLM, eliminating the CUDA dependency that locks most enterprises into NVIDIA's ecosystem.
"SambaNova's platform is differentiated, built for a market where inference has become foundational to enterprise and industry transformation," said Martín Escobari, Co-President at General Atlantic.
Framework #1: Enterprise AI Inference Vendor Selection Matrix
Not every inference architecture fits every workload. Here's how the five major inference platforms compare across the dimensions that matter for enterprise procurement:
| Dimension | NVIDIA (Blackwell + Groq LPX) | Cerebras (WSE-3) | SambaNova (SN50 RDU) | OpenAI Jalapeño (Custom ASIC) | Cloud API (OpenAI/Anthropic/Google) |
|---|---|---|---|---|---|
| Primary Strength | Versatility (train + infer) | Single-chip massive models | Multi-model agentic inference | Cost-optimized token generation | Zero infrastructure management |
| Best For | Mixed training + inference | Ultra-large single-model workloads | Regulated multi-model on-prem | High-volume commodity inference | Prototyping, variable demand |
| On-Prem Available? | Yes (DGX) | Yes (CS-3) | Yes (SambaRack) | No (OpenAI internal only) | No |
| Model Switching Speed | Minutes | Minutes | Microseconds | N/A | Milliseconds (API routing) |
| Max Model Size (single node) | ~1T params (8x B200) | ~50T params (WSE-3 cluster) | ~5T params (single RDU node) | Unknown | Unlimited (provider-managed) |
| Power per Rack | 40-70 kW (liquid cooling) | 20-25 kW (air + liquid) | ~20 kW (air-cooled) | N/A | N/A |
| CUDA Dependency | Yes | No | No | No | No |
| Software Ecosystem | Mature (TensorRT, Triton) | Growing (custom SDK) | Moderate (vLLM, SambaNova Suite) | N/A | Mature (API-based) |
| Enterprise Customers | Universal | Government, research | JPMorgan, sovereign clouds | OpenAI only | Universal |
| TCO (4-year, sustained) | High (hardware + power + CUDA talent) | Medium-High | Medium (lower power, fewer nodes) | Low (at OpenAI scale) | Highest (per-token pricing compounds) |
| Regulatory Control | Full (on-prem) | Full (on-prem) | Full (on-prem) | None | None |
Decision logic:
- If you run mixed training + inference workloads → NVIDIA remains the default, but budget for 55-80% of your GPU fleet serving inference by 2027.
- If you're a regulated enterprise (finance, healthcare, government) running multiple models on-prem → SambaNova's RDU architecture deserves a proof-of-concept. JPMorgan's selection validates the use case.
- If you run a single massive model at extreme scale → Cerebras WSE-3 eliminates multi-GPU communication overhead, but availability is constrained through 2027.
- If inference cost-per-token is your primary concern and you don't need on-prem → Cloud APIs still win on simplicity, but on-prem inference is 18x cheaper per token for sustained workloads.
- If you're building agentic workflows with rapid model switching → SambaNova's microsecond switching and Dell's deskside AI appliances are purpose-built for this pattern.
The On-Prem Inference Migration: 55% and Accelerating
JPMorgan's move isn't an outlier. It's the leading edge of a structural migration.
According to multiple industry reports, 55% of enterprise AI inference now runs on-premise or on-device — up from just 12% in 2023. Meanwhile, only 41% of enterprises still treat public cloud as their primary inference environment.
The economics are driving this reversal. For sustained AI inference workloads, on-premise deployment is up to 62% more cost-effective than public cloud and up to 75% more cost-effective than API-based services over a four-year horizon, according to Lenovo's 2026 TCO analysis. The FinOps Foundation's 2026 State of FinOps Report found that 73% of enterprises reported AI costs exceeding original budget projections — and most of that overshoot is inference, not training.
But cost is only half the story. The regulatory environment is closing the window on cloud-only inference:
- EU AI Act (August 2026): High-risk AI systems in banking, insurance, and HR must be auditable, transparent, and robust. On-prem inference simplifies every audit trail.
- Data localization laws: By end of 2026, 35% of countries will have established regional AI infrastructure requirements.
- GDPR enforcement escalation: High-profile data exposure incidents involving cloud AI APIs have made compliance teams allergic to sending sensitive data through third-party endpoints.
SambaNova CEO Rodrigo Liang put it bluntly: "Having JPMorgan Chase decide they're going to use SambaNova for their inference solution sends a message to the banking industry that it's time not to completely depend on cloud services. These banks want heterogeneous infrastructure."
Framework #2: Enterprise On-Prem AI Inference Readiness Assessment
Before committing to on-prem inference infrastructure, score your organization across these 10 dimensions. Each is rated 1-5 (1 = not ready, 5 = fully ready). A score below 30 means cloud-first remains the right default. A score of 30-40 means hybrid deployment. Above 40, you're ready for dedicated on-prem inference.
Infrastructure Readiness
1. Data Center Capacity (1-5)
- 1: No available rack space or power budget
- 3: Space available but requires cooling upgrades
- 5: Rack space, power (20+ kW per rack), and cooling (air or liquid) ready
2. Network Architecture (1-5)
- 1: No high-bandwidth internal network for model serving
- 3: 25-100 Gbps backbone, adequate for single-model workloads
- 5: 400 Gbps+ fabric optimized for distributed inference with low-latency interconnects
3. Operations Team (1-5)
- 1: No ML infrastructure expertise on staff
- 3: Cloud ML ops team that could retrain for on-prem
- 5: Dedicated ML platform engineering team with hardware deployment experience
Workload Profile
4. Inference Volume Predictability (1-5)
- 1: Highly variable, spiky demand (cloud elasticity critical)
- 3: Predictable base load with seasonal spikes
- 5: Sustained, high-volume inference with <20% demand variance
5. Model Count and Switching Frequency (1-5)
- 1: Single model, rarely updated
- 3: 5-10 models, weekly updates
- 5: 20+ models with real-time switching requirements (agentic workflows)
6. Latency Requirements (1-5)
- 1: Batch processing, latency tolerant (>5s acceptable)
- 3: Near-real-time (<500ms), standard for chatbots
- 5: Ultra-low latency (<50ms) for trading, real-time decisioning, or multi-step agents
Regulatory and Security
7. Data Sovereignty Requirements (1-5)
- 1: No regulatory constraints on data location
- 3: Industry guidelines recommend local processing
- 5: Legal mandate (EU AI Act, GDPR, financial regulators) requiring on-prem data processing
8. Audit Trail Requirements (1-5)
- 1: No inference logging requirements
- 3: Basic request/response logging sufficient
- 5: Full model versioning, input/output capture, explainability, and regulatory audit support
Economic
9. Current Cloud AI Spend (1-5)
- 1: <$100K/year on inference APIs (cloud is cheaper at this scale)
- 3: $500K-$2M/year (approaching on-prem breakeven)
- 5: >$5M/year on inference (on-prem ROI is significant — potentially 62-75% savings)
10. Budget Horizon (1-5)
- 1: Quarter-to-quarter budgeting, no multi-year commitment possible
- 3: Annual budgeting with 2-year visibility
- 5: 3-5 year infrastructure capital planning approved
Scoring Guide
| Score | Recommendation |
|---|---|
| 10-20 | Stay cloud-first. Your volume, team, and infrastructure aren't ready for on-prem inference economics to justify the capital outlay. |
| 21-30 | Hybrid approach. Run predictable base-load inference on-prem (consider Dell's deskside appliances for departmental workloads), burst to cloud. |
| 31-40 | On-prem primary. Evaluate SambaNova, Cerebras, or NVIDIA DGX for dedicated inference infrastructure. Run a 90-day POC with your top 3 models. |
| 41-50 | Full on-prem inference. You're JPMorgan. Build a heterogeneous inference fleet with multiple architectures, negotiate enterprise agreements, and plan for 3-5 year capacity. |
What This Means for Your 2027 Infrastructure Budget
SambaNova's raise isn't just a funding story. It's a market signal with three concrete implications for enterprise AI strategy:
1. Inference infrastructure is now a separate budget line. If your AI budget doesn't distinguish between training and inference spend, you're flying blind on 55-80% of your costs. Split them. Track them. Optimize them independently.
2. NVIDIA is no longer the only enterprise-grade option. SambaNova, Cerebras, Qualcomm/Modular, and OpenAI's custom silicon are all viable alternatives for inference-specific workloads. The CUDA lock-in that defined the last decade is breaking. Budget for at least one non-NVIDIA POC in 2027.
3. On-prem inference is the compliance play, not the legacy play. With the EU AI Act, expanding data localization laws, and the economic case for sustained workloads, the "everything in the cloud" era for AI inference is ending. The enterprises that build heterogeneous, on-prem inference capabilities now will have a structural advantage when regulations tighten further.
SambaNova CEO Liang described three customer categories: sovereign clouds (governments building national AI infrastructure), enterprises in regulated industries, and neo-cloud providers offering inference-as-a-service. All three are growing. SoftBank is the first SN50 deployment partner, with customer shipments beginning in H2 2026.
The inference war has begun. The winners won't be the companies that trained the best model — they'll be the ones that can run it fastest, cheapest, and closest to where the data lives.
Rajesh Beri is Head of AI Engineering at Zscaler, where he builds AI solutions for enterprise security, sales, and operations.
