SWE-bench Documentation — Evaluating Coding Agents on Real GitHub Issues
by SWE-bench Team (Princeton NLP)
Run the benchmark every coding-agent launch quotes, and learn what the number means.
Overview
The site is organized as Installation, User Guides, Reference and an FAQ. The user guides cover a quickstart, the evaluation procedure, Docker configuration, the datasets, and building a RAG dataset from the benchmark's repositories; the reference documents the harness and inference APIs. Evaluation is fully containerized — a Docker image per task instance, introduced in June 2024 — which is what makes runs reproducible across machines, and cloud evaluation via Modal is now supported for teams without local capacity. Four dataset variants are documented: the full SWE-bench, SWE-bench Lite, SWE-bench Verified (500 problems that professional software engineers confirmed are actually solvable), and SWE-bench Multimodal, which adds JavaScript repositories whose issues contain images and targets front-end work such as UI, diagramming and data visualization; the repository also ships SWE-bench Multilingual. As of 1 September 2026 Multimodal v2 is fully open source with 480 tasks runnable locally. The CLI is a single entry point, for example `swebench eval verified -p <predictions> --run-id <id> -j <workers>`, with additional commands for inference generation, image management, reporting and cloud submission via Hugging Face. The project is maintained by the Princeton NLP team behind the ICLR 2024 oral paper, has roughly 5.9k GitHub stars and 973 forks, and welcomes contributions from NLP, ML and software engineering.
At a Glance
- Topic
- Agentic
- Level
- Advanced
- Format
- Documentation
- Cost
- Free
- Duration
- ~2 hours to read the guides; a full SWE-bench Verified run is several hours of containerized compute
- Provider
- SWE-bench Team (Princeton NLP)
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Install the harness and run `swebench eval verified` against your own prediction file
- ✓Choose the right dataset variant: full, Lite, Verified, Multimodal or Multilingual
- ✓Understand containerized evaluation — a Docker image per task instance and why that matters
- ✓Run evaluations on Modal in the cloud when local Docker capacity is not available
- ✓Build a RAG dataset from the benchmark repositories for retrieval experiments
- ✓Generate model patches using the inference scripts and the documented harness API
- ✓Read resolve rates critically, including why the 500 human-verified tasks exist at all
Highlights
- •This is the benchmark every coding-agent release is scored on — reading the harness is how you learn what those percentages actually measure
- •Multimodal v2 became fully open source on 1 September 2026, with 480 tasks runnable locally
- •Containerized per-task Docker evaluation makes results reproducible; Modal support covers cloud runs
- •~5.9k stars and 973 forks, maintained by the Princeton NLP group behind the ICLR 2024 oral paper
- •Documents a RAG-dataset path as well, so the repo is useful for retrieval work and not only agent scoring
Who It’s For
Best For
- ✓Teams building or fine-tuning coding agents who need a defensible, reproducible evaluation
- ✓Engineers auditing vendor SWE-bench claims before acting on a leaderboard
- ✓Researchers producing new agent scaffolds or software-engineering training data
- ✓Platform teams standing up an internal agent benchmark harness
Prerequisites
- •Comfortable with Docker, Python packaging and long-running CLI jobs
- •Familiarity with git patches and diffs, and how a test suite decides pass or fail
- •An agent or model that can already emit patch predictions to score
- •Enough disk and CPU headroom for per-task container images
FAQ
What is SWE-bench Documentation — Evaluating Coding Agents on Real GitHub Issues?
The official documentation for SWE-bench, the benchmark that gives a language model a real repository and a real GitHub issue and scores whether its patch makes the project's tests pass. It is written for engineers who want to run the harness themselves — install it, evaluate their own agent's predictions, and read resolve rates critically instead of trusting a vendor's leaderboard screenshot.
Is SWE-bench Documentation — Evaluating Coding Agents on Real GitHub Issues free?
SWE-bench Documentation — Evaluating Coding Agents on Real GitHub Issues is free to access.
What level is SWE-bench Documentation — Evaluating Coding Agents on Real GitHub Issues for?
SWE-bench Documentation — Evaluating Coding Agents on Real GitHub Issues is aimed at a advanced audience. Recommended background: Comfortable with Docker, Python packaging and long-running CLI jobs, Familiarity with git patches and diffs, and how a test suite decides pass or fail, An agent or model that can already emit patch predictions to score, Enough disk and CPU headroom for per-task container images.
How long does SWE-bench Documentation — Evaluating Coding Agents on Real GitHub Issues take?
Expect roughly ~2 hours to read the guides; a full SWE-bench Verified run is several hours of containerized compute. Most learners work through it at their own pace.
What will I learn from SWE-bench Documentation — Evaluating Coding Agents on Real GitHub Issues?
You'll learn: Install the harness and run `swebench eval verified` against your own prediction file; Choose the right dataset variant: full, Lite, Verified, Multimodal or Multilingual; Understand containerized evaluation — a Docker image per task instance and why that matters; Run evaluations on Modal in the cloud when local Docker capacity is not available; Build a RAG dataset from the benchmark repositories for retrieval experiments; Generate model patches using the inference scripts and the documented harness API; Read resolve rates critically, including why the 500 human-verified tasks exist at all.
Topics
Sources
This page was written from 2 sources, 1 on domains other than swebench.com.