LiteLLM
by BerriAI
Open-source AI gateway: 100+ LLM APIs behind one OpenAI-compatible endpoint, with cost tracking and guardrails
LiteLLM is an open-source AI gateway and LLM proxy that puts more than 100 model providers behind a single OpenAI-compatible API, adding per-key cost tracking, budgets, rate limits, guardrails and logging. It is aimed at platform teams who need to give hundreds of developers governed access to many models without wiring each provider SDK by hand.
LiteLLM, built by Y Combinator-backed BerriAI, is an open-source AI gateway that normalises more than 100 LLM providers — OpenAI, Azure OpenAI, AWS Bedrock, Google Vertex AI, Anthropic, NVIDIA NIM, self-hosted vLLM and others — behind a single OpenAI-compatible interface, so application code targets one API surface regardless of which model serves the request. Around that translation layer it adds the controls platform teams actually need in production: virtual keys, per-key/user/team/organisation budgets and RPM/TPM limits, tag-based spend tracking, guardrails, load balancing and fallbacks, request logging to S3 and Prometheus metrics. The repository was created in July 2023 and has grown to roughly 59,200 GitHub stars, 11,600 forks and about 5,180 open issues, with commits landing daily. Its most consequential recent change is architectural: through 2026 BerriAI migrated the hot path from Python to a Rust core exposed via a Python SDK, staged route by route (OCR first, then /v1/messages, then /chat/completions) rather than shipped as a breaking v2. BerriAI's published benchmarks put the Rust gateway at ~0.05 ms per-request overhead and 6,782 req/s at 31.7 MB peak memory versus ~7.5 ms, 453 req/s and 358.9 MB for the Python proxy — roughly 15x throughput on 11x less memory — while noting the figure measures the forwarding path only. A paid Enterprise tier adds SSO/SCIM, OIDC/JWT, RBAC, audit logs, air-gapped and multi-region deployment and 24/7 SLA support, and is certified SOC 2 Type 2 and ISO 27001. LiteLLM cites Netflix, Okta, Ramp, NASA, Zurich and Cloudera as users, and is distributed via the AWS and Azure marketplaces, with Vercel adding support in April 2026.
The platform engineering lead who has to give hundreds of internal developers and dozens of applications governed, budgeted, audited access to many LLM providers without writing a bespoke proxy.
One OpenAI-compatible endpoint in front of 100+ providers, with per-key spend tracking, budgets and audit logs — self-hostable for free, air-gappable on the Enterprise tier.
At a Glance
- Category
- Infrastructure & Cloud
- Pricing
- Free, Open-source, Subscription, Contact for pricing
- Target Market
- CTOs, Platform Engineers, Enterprise Developers, MLOps Engineers, FinOps Leads
- Deployment
- Open-source, Self-hosted, Hybrid, Multi-cloud
- Founded
- 2023
- Customers
- Publicly named users include Netflix, Okta, Ramp, NASA, Zurich and Cloudera; ~59,200 GitHub stars and 11,600 forks
Key Features
- ✓Unified OpenAI-compatible API
Calls 100+ providers including Bedrock, Azure, Vertex AI, Anthropic and NVIDIA NIM through one request format and response shape.
- ✓Virtual keys, budgets and rate limits
Issues scoped keys with RPM/TPM ceilings and hard spend budgets per key, user, team or organisation.
- ✓Spend tracking and tagging
Attributes token cost back to the team, application or tag that incurred it, which is what makes AI chargeback possible.
- ✓Rust core with Python SDK
Hot path rewritten in Rust for ~0.05 ms overhead and 6,782 req/s, migrated route by route with no breaking version bump.
- ✓Guardrails and logging
Applies request guardrails and writes audit-grade logs to S3 with Prometheus metrics for the existing observability stack.
- ✓Enterprise identity and deployment controls
SSO/SCIM, OIDC/JWT, RBAC, key rotation, secret managers, air-gapped install and multi-region control plane on the paid tier.
Capabilities
Use Cases
- •Central AI gateway for an enterprise
One governed endpoint serves every internal app, so provider credentials never leave the platform team's control.
- •LLM cost chargeback
Tag-based spend tracking attributes token cost to the owning team, turning an opaque AI bill into per-team accountability.
- •Provider failover during an outage
Load balancing and fallbacks reroute traffic to a second provider or region when a primary model endpoint degrades.
- •Air-gapped deployment in a regulated bank
Enterprise air-gapped install keeps prompts and completions entirely inside the customer's own network boundary.
- •Migrating applications between models
Because every provider speaks the same request format, swapping a model becomes a config change rather than a code change.
Ideal For
Best For
- ✓Platform teams standardising multi-provider LLM access behind one API across many internal applications
- ✓Finance and FinOps visibility into AI spend, broken down by key, user, team, organisation or tag
- ✓Regulated or air-gapped environments that must run the gateway inside their own infrastructure
- ✓Avoiding provider lock-in with load balancing and automatic fallbacks between models and regions
- ✓Teams already emitting Prometheus metrics and S3 logs who want LLM traffic in the same observability stack
Not Ideal For
- ✗Single-application teams calling one provider — a direct SDK call avoids an extra network hop and an extra service to operate
- ✗Very high-throughput workloads still on the Python path: practitioners report P99 latency collapsing from seconds to 90+ seconds past roughly 300-500 RPS, and LiteLLM's own docs prescribe worker recycling to contain memory growth
- ✗Organisations without DevOps capacity — third-party analyses put self-hosted setup at 2-4 weeks plus 10-20 hours a month of maintenance, on top of infrastructure cost
- ✗Teams that need vendor-backed security guarantees on the free tier: the March 2026 PyPI supply-chain compromise and the absence of published patch SLAs on open source argue for the paid tier or a hardened alternative
Integrations
Deployment
Market & Ratings
Publicly named users include Netflix, Okta, Ramp, NASA, Zurich and Cloudera; ~59,200 GitHub stars and 11,600 forks
Market Analysis
Pros
- ✓Unmatched provider breadth and a genuine de facto standard — ~59,200 GitHub stars, 11,600 forks, daily commits, and marketplace distribution on both AWS and Azure
- ✓The Rust core materially changes the performance story: BerriAI's benchmarks show ~0.05 ms overhead and 6,782 req/s at 31.7 MB versus ~7.5 ms, 453 req/s and 358.9 MB for the Python proxy
- ✓Free and self-hostable with a real paid tier behind it — SOC 2 Type 2, ISO 27001, SSO/SCIM, RBAC, audit logs, air-gapped install and 24/7 SLAs
- ✓Named production users at serious scale (Netflix, Okta, NASA, Zurich, Cloudera) and first-party Vercel support since April 2026
Cons
- ✗Documented memory leaks and CPU spikes under concurrency — GitHub issue #15128 describes containers consuming all 12 GB and being OOM-killed, and LiteLLM's own troubleshooting docs prescribe recycling workers after a fixed request count to mitigate it
- ✗The Python path degrades badly under load: practitioners report the gateway falling apart around 400-500 RPS with P99 latency going from seconds to 90+ seconds, and slow imports adding 3+ seconds to serverless cold starts
- ✗In March 2026 two official PyPI releases (1.82.7, 1.82.8) shipped a credential stealer — the top Hacker News thread drew 938 points and 500 comments, and Mercor later attributed a breach to the compromise
- ✗About 5,180 open GitHub issues, and Kong's competitive benchmark reports 859% higher throughput and 86% lower latency for its own gateway plus the absence of SLSA Level 3 build attestation or published vulnerability-patching SLAs
- ✗The Rust migration is incremental, not complete — custom Python plugins run in a sidecar and long-tail providers remain on the Python path, so the headline benchmark does not apply to every route or workload
Pricing
Open Source
$0
- ✓100+ provider support
- ✓Virtual keys and budgets
- ✓Load balancing and fallbacks
- ✓Self-hosted, community support
Enterprise
Contact for pricing
- ✓SSO + SCIM, OIDC/JWT, RBAC
- ✓Audit logs on every request
- ✓Air-gapped and multi-region deployment
- ✓24/7 support with response-time SLAs
- ✓SOC 2 Type 2 and ISO 27001
The open-source gateway is free to license but not free to run: a third-party analysis by competitor TrueFoundry puts real total cost of ownership at roughly $2,000-$3,500 a month once infrastructure, DevOps labour and monitoring are counted, and reports Enterprise tiers around $250/month for the basic package and roughly $30,000 a year for premium — figures LiteLLM does not publish itself, so treat them as indicative. LiteLLM offers a 30-day self-hosted trial key with no credit card, and quotes Enterprise through sales.
Security & Compliance
Connect
Sources
This page was written from 7 sources, 6 on domains other than litellm.ai.
Stay Ahead of the Curve
Weekly enterprise AI insights for technology leaders. No spam, no vendor pitches—unsubscribe anytime.
SubscribeRelated Products
CIQ Fuzzball
Sovereign AI and HPC orchestration: train, fine-tune and serve models on infrastructure you control
Crusoe Cloud
Vertically integrated AI cloud — own the power, own the datacentre, rent the GPUs by the minute
Nexthop AI
Ethernet switches and network software co-designed for AI clusters, built for hyperscalers and NeoClouds
InsightFinder
Predictive reliability for AI agents and IT estates — unsupervised anomaly detection that flags incidents before they land