LiteLLM Documentation — AI Gateway and Python SDK
by BerriAI
One OpenAI-format call to 100+ providers, plus the self-hosted gateway with virtual keys, budgets and fallbacks.
Overview
LiteLLM is the de facto open-source AI gateway, and its documentation is split into two tracks serving different jobs: the Python SDK, for application developers who want one completion() call reaching 100+ providers in OpenAI input/output format, and the Proxy Server — documented as the LLM Gateway — for platform teams who need a central service in front of every model their organisation uses. The SDK docs cover the router's retry and fallback logic across deployments, application-level load balancing, callback-based observability into Langfuse, MLflow and Lunary, and an exception-mapping layer that normalises each provider's error taxonomy onto OpenAI's. The proxy docs cover self-hosted deployment, virtual keys with per-key budgets and rate limits, multi-tenant cost tracking, guardrails, and an admin dashboard UI. A compatibility matrix records which of /chat/completions, /embeddings, /images, /audio, /batches and /rerank each provider actually supports, which is considerably more useful than the headline provider count. The project sits at 55.3k GitHub stars and has migrated its core to Rust while keeping the Python interface. Read the March 2026 security advisory before deploying: a compromised Trivy action in CI leaked the PyPI publish token, and versions 1.82.7 and 1.82.8 shipped a litellm_init.pth credential stealer that ran on interpreter start. Pin to 1.82.6 or earlier, or 1.83.0 and later.
At a Glance
- Topic
- Frameworks
- Level
- Intermediate
- Format
- Documentation
- Cost
- Free
- Duration
- ~2-3 hours for the SDK quickstart plus a working proxy deployment; ongoing reference thereafter
- Provider
- BerriAI
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Call 100+ LLM providers through one OpenAI-compatible request and response format
- ✓Configure router fallbacks and retries across multiple deployments of the same model
- ✓Deploy the proxy as a self-hosted gateway with a multi-tenant admin dashboard
- ✓Issue virtual keys carrying per-key budgets, spend tracking and rate limits
- ✓Wire observability callbacks into Langfuse, MLflow, Lunary and other logging platforms
- ✓Map provider-specific exceptions onto a single standardized OpenAI-style error taxonomy
- ✓Read the compatibility matrix to check endpoint support before committing to a provider
- ✓Apply guardrails and per-project isolation when serving many internal teams
Highlights
- •55.3k GitHub stars and the default answer for teams who need one gateway across providers — the docs are effectively the spec for that pattern
- •Cleanly split into SDK and Proxy tracks, so an app developer and a platform engineer each read half the site rather than filtering one narrative
- •The per-provider endpoint compatibility matrix is unusually honest about what actually works where, instead of a flat '100+ providers' claim
- •Documents the March 2026 supply-chain compromise in its own security advisory: versions 1.82.7 and 1.82.8 carried a credential stealer for roughly 40 minutes on PyPI. Pin to 1.82.6 or earlier or 1.83.0+, and treat this as required reading, not a footnote
- •Core migrated to Rust in 2026 while keeping the Python SDK surface; stable Docker tags now follow 12-hour load tests
Who It’s For
Best For
- ✓AI engineers who need provider failover without writing their own abstraction layer
- ✓ML platform teams standing up a shared internal gateway with budgets and virtual keys
- ✓Agent developers who want to swap models per task without rewriting call sites
- ✓Anyone doing per-team or per-project LLM cost attribution across several vendors
Prerequisites
- •Working Python and experience calling at least one LLM provider API directly
- •Docker and basic Kubernetes for anything beyond the SDK — the proxy ships as a container
- •Enough supply-chain security awareness to pin dependency versions and rotate credentials
FAQ
What is LiteLLM Documentation — AI Gateway and Python SDK?
The reference for the most widely deployed open-source AI gateway, written for two audiences at once: application developers who want a single call that reaches any provider, and platform teams who need a central service with cost tracking, virtual keys and failover in front of every model their organisation uses. Read it before you write your own provider abstraction layer.
Is LiteLLM Documentation — AI Gateway and Python SDK free?
LiteLLM Documentation — AI Gateway and Python SDK is free to access.
What level is LiteLLM Documentation — AI Gateway and Python SDK for?
LiteLLM Documentation — AI Gateway and Python SDK is aimed at a intermediate audience. Recommended background: Working Python and experience calling at least one LLM provider API directly, Docker and basic Kubernetes for anything beyond the SDK — the proxy ships as a container, Enough supply-chain security awareness to pin dependency versions and rotate credentials.
How long does LiteLLM Documentation — AI Gateway and Python SDK take?
Expect roughly ~2-3 hours for the SDK quickstart plus a working proxy deployment; ongoing reference thereafter. Most learners work through it at their own pace.
What will I learn from LiteLLM Documentation — AI Gateway and Python SDK?
You'll learn: Call 100+ LLM providers through one OpenAI-compatible request and response format; Configure router fallbacks and retries across multiple deployments of the same model; Deploy the proxy as a self-hosted gateway with a multi-tenant admin dashboard; Issue virtual keys carrying per-key budgets, spend tracking and rate limits; Wire observability callbacks into Langfuse, MLflow, Lunary and other logging platforms; Map provider-specific exceptions onto a single standardized OpenAI-style error taxonomy; Read the compatibility matrix to check endpoint support before committing to a provider; Apply guardrails and per-project isolation when serving many internal teams.
Topics
Sources
This page was written from 4 sources, 2 on domains other than docs.litellm.ai.