Tensormesh

Tensormesh is an inference platform from the maintainers of LMCache that persists and reuses the key-value cache LLM servers normally throw away after every request. It targets AI platform and infrastructure teams running agentic, RAG or long-conversation workloads, where the same system prompt, document set or chat history is reprocessed on every call and dominates the GPU bill.

Tensormesh commercialises LMCache, the open-source KV-caching engine created and maintained by its co-founders, with roughly 11.7k GitHub stars, 1.8k forks and an Apache 2.0 licence. The insight is that a standard LLM server discards its key-value cache after each query, so every request recomputes the same system prompts, retrieved documents and conversation history from scratch; CEO Junchen Jiang describes it as an analyst who forgets everything after each question. LMCache instead persists that cache and tiers it across GPU, CPU, local disk and remote backends, sharing it across sessions and replicas, and integrates with vLLM, SGLang, TensorRT, AWS SageMaker and Oracle OCI Data Science. The commercial product, Tensormesh Inference, became generally available on 27 May 2026 alongside a $20 million funding round from AMD Ventures, CoreWeave, NVIDIA's NVentures, Valley Capital Partners and Laude Ventures, taking total funding to $24.5 million after a $4.5 million seed led by Laude in October 2025. It ships in two shapes: serverless inference behind an OpenAI-compatible API across 15+ open-weight models from Qwen, OpenAI, DeepSeek, Google, Moonshot, Z.ai, Mistral and MiniMax with 128K–1M context windows, and reserved deployments with dedicated capacity and custom containers. The commercial hook is that cached input tokens are billed at $0, with a dashboard exposing cache hit rates, time-to-first-token, inter-token latency and throughput. Tensormesh claims up to 10x reductions in latency and GPU spend, with hit rates frequently above 70% in tuned deployments. An on-premises product, Tensormesh Operator, is announced but not yet shipped.

Category

Infrastructure & Cloud

Pricing

Usage-based, Subscription

Target Market

CTOs, VP Engineering, AI Infrastructure Engineers, ML Platform Teams, Enterprise Developers

Deployment

Cloud-only, API-based, Open-source

Headquarters

Foster City, California, United States

Team Size

11-50

• ✓
  Persistent cross-request KV caching

  Stores and reuses the key-value cache between requests and sessions instead of discarding it, so repeated context is never recomputed

• ✓
  Tiered cache offload

  Spills cache across GPU, CPU, local disk and remote backends so reuse is not bounded by GPU memory on a single replica

• ✓
  Cached tokens billed at $0

  Input tokens served from cache are not charged, so cost per request falls as context reuse in the workload rises

• ✓
  OpenAI-compatible serverless API

  Drop-in endpoint across 15+ open-weight models with 128K to 1M context windows, so migration is a base-URL change rather than a rewrite

• ✓
  Reserved GPU capacity

  Dedicated H200 replicas at a published hourly rate with custom containers and beta autoscaling for predictable production performance

• ✓
  Cache and latency observability

  Dashboard reports cache hit rate, token-level cost breakdown, time-to-first-token, inter-token latency and throughput so savings are verifiable

• •
  Cutting agent loop cost

  An agent replaying a long tool-call log each step reuses the cached prefix, so only the new turn is billed and computed

• •
  High-volume RAG serving

  Shared document chunks stay cached across users, dropping time-to-first-token and removing repeated prefill work from the GPU bill

• •
  Long-context document assistants

  A 256K-token contract stays cached across a whole review session instead of being re-ingested on every follow-up question

• •
  Migrating off a general inference provider

  Teams point an existing OpenAI-compatible client at Tensormesh and measure hit rate and spend before committing reserved capacity

• •
  Self-hosted vLLM cost tuning

  Platform teams adopt open-source LMCache first for tiered cache offload, then move to the hosted platform if operating it proves costly

Pricing is published per million tokens and varies by model: Qwen3 30B at $0.15 input / $0.60 output, OpenAI gpt-oss-20b at $0.07 / $0.28, Mistral Devstral-2 123B at $0.50 / $2.00, rising to $0.60 / $3.60 for a 397B Qwen model. Cached input tokens cost $0.00, which is the whole commercial pitch, so real cost depends entirely on your cache hit rate rather than a headline rate. Reserved H200 capacity is $2.50/GPU-hour, roughly $1,825/month for one continuously running replica. No free tier, trial credits or published enterprise tier are listed.

This page was written from 6 sources, 3 on domains other than tensormesh.ai.

1. 1.tensormesh.ai — tensormesh.aivendor
2. 2.tensormesh.ai — pricingvendor
3. 3.tensormesh.ai — tensormesh raises 20m launches inference platformvendor
4. 4.techcrunch.com — tensormesh raises 4 5m to squeeze more inference out of ai s
5. 5.pulse2.com — tensormesh 20 million raised to scale kv caching infrastruct
6. 6.github.com — LMCache