MTEB — Massive Text Embedding Benchmark
by Embeddings Benchmark community
The framework and leaderboard the embedding field uses to compare retrievers, rerankers and encoders — and to score your own.
Overview
MTEB is the open-source Python framework and public leaderboard that the embedding-model field uses to compare retrievers, rerankers and encoders. It is maintained by the embeddings-benchmark community on GitHub (~3.4k stars, 4,500+ commits, active issue and PR traffic) with an interactive leaderboard hosted on Hugging Face Spaces. The documentation is organised into installation, a get-started usage track, a CLI reference, an overview of tasks and benchmarks, a full API reference, and a contributing guide for adding your own tasks or models. Evaluation spans more than a thousand tasks across families including retrieval, reranking, classification, clustering, pair classification, semantic textual similarity, summarisation scoring and bitext mining, covering the original English MTEB, the multilingual MMTEB expansion, and image and audio benchmarks. The v2.0.0 release is a genuine rewrite: the old MTEB class is replaced by a single mteb.evaluate() call with an overwrite_strategy argument; a ResultCache pulls previously published scores from a remote cache so you do not re-run models others have already scored; explicit CrossEncoderProtocol and SearchProtocol interfaces cover rerankers and end-to-end search; model inputs moved to a batched multimodal format carrying text, images and audio; and every task now ships descriptive statistics on sample counts, text lengths and duplicates. A companion 2025 paper documents the continuous-integration, dataset-validation and versioning practices that keep old results reproducible as the benchmark grows.
At a Glance
- Topic
- RAG
- Level
- Intermediate
- Format
- Documentation
- Cost
- Free
- Duration
- ~2-3 hours to install and run your first evaluation; docs are ongoing reference
- Provider
- Embeddings Benchmark community
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Run a full embedding evaluation from either the CLI or the Python API
- ✓Select task subsets and benchmark suites instead of running all 1000+ tasks
- ✓Interpret retrieval, reranking, clustering and STS scores rather than one leaderboard number
- ✓Evaluate cross-encoder rerankers and end-to-end search pipelines through dedicated protocols
- ✓Reuse previously published results via ResultCache instead of recomputing every baseline
- ✓Wrap a custom or private embedding model so MTEB can score it
- ✓Add a new task or dataset and pass the repository's CI validation
- ✓Read per-task descriptive statistics to spot duplicate rows and length skew
Highlights
- •The de facto leaderboard the embedding field cites — model vendors publish MTEB numbers by default
- •v2.0.0 is a real rewrite: one mteb.evaluate() entry point, remote result caching, batched multimodal inputs
- •Covers text, image and audio embeddings in one harness, not just English text retrieval
- •A published maintenance paper documents the CI and versioning that keep older scores reproducible
- •Practitioners criticise it openly — a widely-read Hacker News thread argued in-domain versus out-of-domain variance makes the headline ranking a weak proxy for your corpus, so treat it as a shortlist, not a verdict
Who It’s For
Best For
- ✓RAG engineers choosing an embedding or reranker model for a specific corpus
- ✓Teams that need a reproducible offline evaluation before swapping retrieval models
- ✓Researchers publishing a new embedding model who need comparable numbers
- ✓Anyone auditing whether a vendor's leaderboard claim survives on their own tasks
Prerequisites
- •Working Python, plus pip or uv for installation
- •Familiarity with embeddings, vector search and retrieval metrics such as nDCG
- •A GPU is helpful — running full benchmark suites on CPU is slow
FAQ
What is MTEB — Massive Text Embedding Benchmark?
MTEB is the open-source evaluation harness behind the embedding leaderboard every vendor quotes. It is aimed at RAG and search engineers who need to pick an embedding model or reranker for a specific corpus rather than trust a headline ranking. After working through the docs you can score any encoder, cross-encoder or full retrieval pipeline on chosen task subsets, reuse published baselines instead of recomputing them, and contribute a task built from your own data.
Is MTEB — Massive Text Embedding Benchmark free?
MTEB — Massive Text Embedding Benchmark is free to access.
What level is MTEB — Massive Text Embedding Benchmark for?
MTEB — Massive Text Embedding Benchmark is aimed at a intermediate audience. Recommended background: Working Python, plus pip or uv for installation, Familiarity with embeddings, vector search and retrieval metrics such as nDCG, A GPU is helpful — running full benchmark suites on CPU is slow.
How long does MTEB — Massive Text Embedding Benchmark take?
Expect roughly ~2-3 hours to install and run your first evaluation; docs are ongoing reference. Most learners work through it at their own pace.
What will I learn from MTEB — Massive Text Embedding Benchmark?
You'll learn: Run a full embedding evaluation from either the CLI or the Python API; Select task subsets and benchmark suites instead of running all 1000+ tasks; Interpret retrieval, reranking, clustering and STS scores rather than one leaderboard number; Evaluate cross-encoder rerankers and end-to-end search pipelines through dedicated protocols; Reuse previously published results via ResultCache instead of recomputing every baseline; Wrap a custom or private embedding model so MTEB can score it; Add a new task or dataset and pass the repository's CI validation; Read per-task descriptive statistics to spot duplicate rows and length skew.
Topics
Sources
This page was written from 4 sources, 3 on domains other than github.com.