RAGFine-TuningModels

Token Pooling-Aware Training for Multi-Vector Embedding Models

by Hugging Face (Tom Aarsen, Sentence Transformers)

AdvancedTutorialFree~45 min read; the training run takes up to ~4 hours on an A10G-class GPU

Train ColBERT-style retrievers that keep their accuracy after you cut document vectors by 75% with token pooling.

Start LearningAdded Nov 17, 2026 · Updated Nov 17, 2026

Overview

Token Pooling-Aware Training for Multi-Vector Embedding Models was published on the Hugging Face blog on November 11, 2026 by Tom Aarsen, who maintains Sentence Transformers. It starts from the main cost of multi-vector retrieval: storing one vector per document token. The article then introduces HierarchicalTokenPooling, which merges similar token vectors with Ward clustering, and shows three ways to set a budget (pool_factor, max_tokens, or a distance_threshold calibrated to a target vector count). The core section adds a loss wrapper, MultiVectorTokenPoolingLoss, that averages an unpooled retrieval loss with a loss on documents pooled to about a quarter of their vectors. A straight-through estimator carries gradients through the discrete merge decisions. Optional CUDA and Triton kernels speed up MaxSim scoring and clustering. The experiment trains two ModernBERT-base models with 128-dimensional token projections on 120,000 MS MARCO triplets, identical except for the loss, and evaluates both on NanoBEIR. The pooling-aware model scores higher at every budget: at pool_factor=4 it keeps 97.8% of its unpooled nDCG@10 against 94.6% for the baseline, with 74.8% fewer vectors, and the gap widens to 5.78 points at 4 vectors per document. Both models, the training and evaluation scripts, and an sbert.net README are public, and the article covers combining pooling with quantization.

At a Glance

Topic
RAG
Level
Advanced
Format
Tutorial
Cost
Free
Duration
~45 min read; the training run takes up to ~4 hours on an A10G-class GPU
Provider
Hugging Face (Tom Aarsen, Sentence Transformers)
Hands-on
Yes — code/exercises
Certificate
None

What You’ll Learn

  • ✓Why multi-vector retrieval indexes grow with document length and what that costs
  • ✓How to apply HierarchicalTokenPooling with pool_factor, max_tokens or a distance threshold
  • ✓How to calibrate an adaptive pooling threshold to hit a target average vector count
  • ✓How to wrap a cached contrastive loss in MultiVectorTokenPoolingLoss for pooling-aware training
  • ✓How a straight-through estimator passes gradients through discrete Ward clustering merges
  • ✓How to enable optional MaxSim and Ward clustering kernels and run training with hf jobs and uv
  • ✓How to compare pooled and unpooled retrieval quality fairly on NanoBEIR with nDCG@10

Highlights

  • •A matched ablation: the two models differ only in the loss, with no hyperparameter search, so the gain is attributable
  • •Full results table across eleven pooling budgets, with gains from +0.55 to +5.78 nDCG@10 points
  • •Released both checkpoints (tomaarsen/mve-ModernBERT-base-token-pooling and its baseline) plus training and eval scripts in the sentence-transformers repo
  • •Written by the Sentence Transformers maintainer, so the API shown is the library's own, documented on sbert.net

Who It’s For

Best For

  • ✓Engineers running ColBERT-style or late-interaction retrieval who need smaller indexes
  • ✓RAG teams deciding between single-vector and multi-vector embeddings on storage cost
  • ✓Practitioners already training embedding models with Sentence Transformers

Prerequisites

  • •Familiarity with multi-vector retrieval and MaxSim late-interaction scoring
  • •Experience training embedding models with contrastive losses in Sentence Transformers
  • •A CUDA GPU (A10G-class in the example) for the default kernels, or patience with the CPU fallback

FAQ

What is Token Pooling-Aware Training for Multi-Vector Embedding Models?

A Sentence Transformers tutorial by maintainer Tom Aarsen on training multi-vector (ColBERT-style) embedding models that survive token pooling. It is for engineers running late-interaction retrieval who want smaller indexes without losing nDCG. You come away able to configure pooling budgets, calibrate adaptive pooling and train a pooling-aware retriever.

Is Token Pooling-Aware Training for Multi-Vector Embedding Models free?

Token Pooling-Aware Training for Multi-Vector Embedding Models is free to access.

What level is Token Pooling-Aware Training for Multi-Vector Embedding Models for?

Token Pooling-Aware Training for Multi-Vector Embedding Models is aimed at a advanced audience. Recommended background: Familiarity with multi-vector retrieval and MaxSim late-interaction scoring, Experience training embedding models with contrastive losses in Sentence Transformers, A CUDA GPU (A10G-class in the example) for the default kernels, or patience with the CPU fallback.

How long does Token Pooling-Aware Training for Multi-Vector Embedding Models take?

Expect roughly ~45 min read; the training run takes up to ~4 hours on an A10G-class GPU. Most learners work through it at their own pace.

What will I learn from Token Pooling-Aware Training for Multi-Vector Embedding Models?

You'll learn: Why multi-vector retrieval indexes grow with document length and what that costs; How to apply HierarchicalTokenPooling with pool_factor, max_tokens or a distance threshold; How to calibrate an adaptive pooling threshold to hit a target average vector count; How to wrap a cached contrastive loss in MultiVectorTokenPoolingLoss for pooling-aware training; How a straight-through estimator passes gradients through discrete Ward clustering merges; How to enable optional MaxSim and Ward clustering kernels and run training with hf jobs and uv; How to compare pooled and unpooled retrieval quality fairly on NanoBEIR with nDCG@10.

Topics

multi-vector-retrievalcolberttoken-poolingsentence-transformersembeddingsrag

Sources

This page was written from 2 sources, 1 on domains other than huggingface.co.

  1. 1.huggingface.co — token poolingvendor
  2. 2.sbert.net — README