Token Pooling-Aware Training for Multi-Vector Embedding Models
by Hugging Face (Tom Aarsen, Sentence Transformers)
Train ColBERT-style retrievers that keep their accuracy after you cut document vectors by 75% with token pooling.
Overview
Token Pooling-Aware Training for Multi-Vector Embedding Models was published on the Hugging Face blog on November 11, 2026 by Tom Aarsen, who maintains Sentence Transformers. It starts from the main cost of multi-vector retrieval: storing one vector per document token. The article then introduces HierarchicalTokenPooling, which merges similar token vectors with Ward clustering, and shows three ways to set a budget (pool_factor, max_tokens, or a distance_threshold calibrated to a target vector count). The core section adds a loss wrapper, MultiVectorTokenPoolingLoss, that averages an unpooled retrieval loss with a loss on documents pooled to about a quarter of their vectors. A straight-through estimator carries gradients through the discrete merge decisions. Optional CUDA and Triton kernels speed up MaxSim scoring and clustering. The experiment trains two ModernBERT-base models with 128-dimensional token projections on 120,000 MS MARCO triplets, identical except for the loss, and evaluates both on NanoBEIR. The pooling-aware model scores higher at every budget: at pool_factor=4 it keeps 97.8% of its unpooled nDCG@10 against 94.6% for the baseline, with 74.8% fewer vectors, and the gap widens to 5.78 points at 4 vectors per document. Both models, the training and evaluation scripts, and an sbert.net README are public, and the article covers combining pooling with quantization.
At a Glance
- Topic
- RAG
- Level
- Advanced
- Format
- Tutorial
- Cost
- Free
- Duration
- ~45 min read; the training run takes up to ~4 hours on an A10G-class GPU
- Provider
- Hugging Face (Tom Aarsen, Sentence Transformers)
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Why multi-vector retrieval indexes grow with document length and what that costs
- ✓How to apply HierarchicalTokenPooling with pool_factor, max_tokens or a distance threshold
- ✓How to calibrate an adaptive pooling threshold to hit a target average vector count
- ✓How to wrap a cached contrastive loss in MultiVectorTokenPoolingLoss for pooling-aware training
- ✓How a straight-through estimator passes gradients through discrete Ward clustering merges
- ✓How to enable optional MaxSim and Ward clustering kernels and run training with hf jobs and uv
- ✓How to compare pooled and unpooled retrieval quality fairly on NanoBEIR with nDCG@10
Highlights
- •A matched ablation: the two models differ only in the loss, with no hyperparameter search, so the gain is attributable
- •Full results table across eleven pooling budgets, with gains from +0.55 to +5.78 nDCG@10 points
- •Released both checkpoints (tomaarsen/mve-ModernBERT-base-token-pooling and its baseline) plus training and eval scripts in the sentence-transformers repo
- •Written by the Sentence Transformers maintainer, so the API shown is the library's own, documented on sbert.net
Who It’s For
Best For
- ✓Engineers running ColBERT-style or late-interaction retrieval who need smaller indexes
- ✓RAG teams deciding between single-vector and multi-vector embeddings on storage cost
- ✓Practitioners already training embedding models with Sentence Transformers
Prerequisites
- •Familiarity with multi-vector retrieval and MaxSim late-interaction scoring
- •Experience training embedding models with contrastive losses in Sentence Transformers
- •A CUDA GPU (A10G-class in the example) for the default kernels, or patience with the CPU fallback
FAQ
What is Token Pooling-Aware Training for Multi-Vector Embedding Models?
A Sentence Transformers tutorial by maintainer Tom Aarsen on training multi-vector (ColBERT-style) embedding models that survive token pooling. It is for engineers running late-interaction retrieval who want smaller indexes without losing nDCG. You come away able to configure pooling budgets, calibrate adaptive pooling and train a pooling-aware retriever.
Is Token Pooling-Aware Training for Multi-Vector Embedding Models free?
Token Pooling-Aware Training for Multi-Vector Embedding Models is free to access.
What level is Token Pooling-Aware Training for Multi-Vector Embedding Models for?
Token Pooling-Aware Training for Multi-Vector Embedding Models is aimed at a advanced audience. Recommended background: Familiarity with multi-vector retrieval and MaxSim late-interaction scoring, Experience training embedding models with contrastive losses in Sentence Transformers, A CUDA GPU (A10G-class in the example) for the default kernels, or patience with the CPU fallback.
How long does Token Pooling-Aware Training for Multi-Vector Embedding Models take?
Expect roughly ~45 min read; the training run takes up to ~4 hours on an A10G-class GPU. Most learners work through it at their own pace.
What will I learn from Token Pooling-Aware Training for Multi-Vector Embedding Models?
You'll learn: Why multi-vector retrieval indexes grow with document length and what that costs; How to apply HierarchicalTokenPooling with pool_factor, max_tokens or a distance threshold; How to calibrate an adaptive pooling threshold to hit a target average vector count; How to wrap a cached contrastive loss in MultiVectorTokenPoolingLoss for pooling-aware training; How a straight-through estimator passes gradients through discrete Ward clustering merges; How to enable optional MaxSim and Ward clustering kernels and run training with hf jobs and uv; How to compare pooled and unpooled retrieval quality fairly on NanoBEIR with nDCG@10.
Topics
Sources
This page was written from 2 sources, 1 on domains other than huggingface.co.