A Survey on Evaluation of LLM-based Agents

A Survey on Evaluation of LLM-based Agents appeared in Findings of ACL 2026 (pages 26690-26714), by Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, Roy Bar-Haim, Arman Cohan and Michal Shmueli-Scheuer of the Hebrew University of Jerusalem, IBM Research and Yale University. It first went on arXiv in March 2025 and was revised in April 2026. Section 2 covers four capabilities: planning and multi-step reasoning, function calling and tool use, self-reflection, and memory. Section 3 reviews benchmarks for web, software engineering, scientific and conversational agents, including Mind2Web, WebArena, SWE-bench Verified, SWE-Lancer and tau-Bench. Section 4 covers generalist agents through GAIA, OSWorld, AppWorld and the Holistic Agent Leaderboard. Section 5 compares benchmarks on data curation, environment, interaction interface, metric and safety. Section 6 reviews development frameworks such as LangSmith, Langfuse, Google Vertex AI, Arize, Galileo, Patronus AI, AgentEvals and Mosaic AI, with a table of their capabilities, and gym-like environments such as MLGym and BrowserGym. Section 7 names two trends, realistic evaluation and live benchmarks, plus gaps in granular evaluation, cost metrics, scaling, safety and separating LLM performance from harness design. Appendix E gives benchmark recommendations. The authors call the survey a snapshot, send readers to primary papers for depth, and maintain a GitHub repository tracking new work.

Topic

Agentic

Level

Intermediate

Format

Paper

Cost

Free

Duration

25 pages including references and appendices; ~2 hours to read, longer if you follow the benchmarks it cites

Provider

Asaf Yehudai et al. (Hebrew University of Jerusalem, IBM Research, Yale University)

Hands-on

No

Certificate

None

• ✓Evaluate agent planning, tool use, self-reflection and memory with benchmarks built for each capability
• ✓Choose among web, software engineering, scientific and conversational agent benchmarks for your application domain
• ✓Compare benchmarks on data curation, static or dynamic environment, interaction interface, metric and safety coverage
• ✓Distinguish final-response evaluation, stepwise evaluation and trajectory-based assessment when designing an agent test plan
• ✓Weigh reference-based against reference-free trajectory evaluation and understand the reliability cost of LLM-based judges
• ✓Compare LangSmith, Langfuse, Vertex AI, Arize, Galileo, Patronus AI, AgentEvals and Mosaic AI by supported capabilities
• ✓Identify gaps in current practice around cost metrics, safety compliance and separating LLM from harness performance

• •Table 1 compares eight representative benchmarks across data, environment, interface, metric and safety; only tau-Bench is marked as incorporating safety constraints
• •Table 2 sets eight evaluation frameworks against six capabilities, including stepwise assessment, monitoring, trajectory assessment and A/B comparisons, though the authors note some capabilities are early
• •It gives concrete difficulty markers, such as SWE-bench Pro's 1,865 human-verified tasks across 41 repositories, where model performance stayed below 25% Pass@1
• •It explains why static benchmarks go stale and how BFCL versions and the SWE-bench family (Verified, Pro) respond as live benchmarks
• •It flags that most benchmarks conflate the backbone LLM with the agent harness, which matters when you read a leaderboard to choose a stack

This page was written from 3 sources, 1 on domains other than aclanthology.org.

1. 1.aclanthology.org — 2026.findings acl.1330vendor
2. 2.aclanthology.org — 2026.findings acl.1330.pdfvendor
3. 3.arxiv.org — 2503.16416