Language Model Evaluation Harness (lm-eval)
by EleutherAI
The reference harness for benchmarking any LLM on the same inputs, prompts and scoring code.
Overview
The Language Model Evaluation Harness, maintained by EleutherAI, is the open-source framework most published LLM benchmark numbers are computed with, and it is the backend behind Hugging Face's Open LLM Leaderboard. Its design premise, stated by EleutherAI directly, is that model performance is often governed by minor implementation details and results rarely transfer between codebases, so the harness supplies one unifying place where any causal language model is tested on the same exact inputs and codebase, with versioned tasks so results stay comparable to prior work. It ships over 60 standard academic benchmarks spanning hundreds of subtasks and variants, and cleanly separates task definitions from model backends: tasks are declared in YAML with Jinja2 prompt templates, while supported backends include Hugging Face Transformers with quantization, vLLM, SGLang, NVIDIA NeMo and Megatron-LM, ONNX Runtime GenAI, local inference servers, and hosted APIs via OpenAI, Anthropic or LiteLLM. Operationally it handles automatic batch-size detection, request caching so re-runs skip already-evaluated samples, multi-GPU evaluation through data and tensor parallelism, steering-vector application, and Weights & Biases or Zeno result logging. A December 2025 refactor reorganised the CLI into run, ls and validate subcommands and moved model backends into optional installs to shrink the base package. Documentation covers the CLI, YAML configuration, the Python API, and guides for adding new tasks and new model types. It is used internally at NVIDIA, Cohere, BigScience, BigCode, Nous Research and Mosaic ML.
At a Glance
- Topic
- Frameworks
- Level
- Intermediate
- Format
- Documentation
- Cost
- Free
- Duration
- ~3-4 hours to a first reproducible benchmark run; reference docs thereafter
- Provider
- EleutherAI
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Run a reproducible multi-benchmark evaluation against any Hugging Face or API-served model
- ✓Write a new benchmark task in YAML with Jinja2 prompt templates and task versioning
- ✓Register a custom model backend so private or in-house models can be evaluated
- ✓Use automatic batch-size detection and request caching to cut evaluation cost
- ✓Evaluate across multiple GPUs using data-parallel and tensor-parallel configurations
- ✓Serve models through vLLM or SGLang to make large benchmark sweeps tractable
- ✓Diagnose why your scores differ from a paper's, starting with prompt formatting and few-shot setup
Highlights
- •Backend for Hugging Face's Open LLM Leaderboard, so your numbers land on the same scale as the leaderboard's
- •13.7k GitHub stars and used internally at NVIDIA, Cohere, BigScience, BigCode, Nous Research and Mosaic ML
- •December 2025 refactor split backends into optional installs and reorganised the CLI into run/ls/validate subcommands
- •One harness spans HF Transformers, vLLM, SGLang, NeMo, Megatron-LM, ONNX Runtime GenAI and hosted APIs
- •Tasks are explicitly versioned, so a benchmark's prompt format is pinned rather than silently drifting between runs
Who It’s For
Best For
- ✓Teams that need to publish or defend a model's benchmark numbers
- ✓Engineers validating that quantization or pruning did not degrade a model
- ✓Researchers comparing a fine-tune against its base model on identical tasks
- ✓Anyone attempting to reproduce a paper's reported evaluation results
Prerequisites
- •Comfortable with Python packaging and running CLI tools in a GPU environment
- •Familiarity with LLM inference basics: tokenization, few-shot prompting, log-likelihood scoring
- •Access to GPUs or a hosted model API for anything beyond small models
FAQ
What is Language Model Evaluation Harness (lm-eval)?
The de facto standard framework for running academic LLM benchmarks reproducibly. It exposes more than 60 standard benchmarks with hundreds of subtasks behind one interface, and runs them identically against Hugging Face Transformers, vLLM, SGLang, NVIDIA NeMo, Megatron-LM or commercial APIs — so a score you compute is comparable to a published one. This is the tooling layer under Hugging Face's Open LLM Leaderboard.
Is Language Model Evaluation Harness (lm-eval) free?
Language Model Evaluation Harness (lm-eval) is free to access.
What level is Language Model Evaluation Harness (lm-eval) for?
Language Model Evaluation Harness (lm-eval) is aimed at a intermediate audience. Recommended background: Comfortable with Python packaging and running CLI tools in a GPU environment, Familiarity with LLM inference basics: tokenization, few-shot prompting, log-likelihood scoring, Access to GPUs or a hosted model API for anything beyond small models.
How long does Language Model Evaluation Harness (lm-eval) take?
Expect roughly ~3-4 hours to a first reproducible benchmark run; reference docs thereafter. Most learners work through it at their own pace.
What will I learn from Language Model Evaluation Harness (lm-eval)?
You'll learn: Run a reproducible multi-benchmark evaluation against any Hugging Face or API-served model; Write a new benchmark task in YAML with Jinja2 prompt templates and task versioning; Register a custom model backend so private or in-house models can be evaluated; Use automatic batch-size detection and request caching to cut evaluation cost; Evaluate across multiple GPUs using data-parallel and tensor-parallel configurations; Serve models through vLLM or SGLang to make large benchmark sweeps tractable; Diagnose why your scores differ from a paper's, starting with prompt formatting and few-shot setup.
Topics
Sources
This page was written from 2 sources, 1 on domains other than github.com.