Arena AI Evaluations
by Arena (formerly LMArena)
Paid model evaluations built on the Arena leaderboard's crowdsourced human votes, for enterprises choosing an LLM
Arena AI Evaluations is the commercial evaluation service from Arena, the company behind the public LMArena model leaderboard. It sells model labs and enterprises detailed analytics on how models perform in real human comparisons, so a team picking a model for an internal use case has more to go on than static benchmark scores.
Arena AI Evaluations is the paid product of Arena (Arena Intelligence Inc.), the San Francisco company spun out of UC Berkeley's LMSYS group, which built Chatbot Arena in 2023 as a research side project. The public site lets anyone enter a prompt, see two anonymous model answers side by side and vote for the better one; those votes feed a Bradley-Terry style leaderboard, with separate arenas for web development, search, vision and coding assistants. Arena introduced AI Evaluations on September 16, 2025, offering enterprises, model labs and developers evaluation services grounded in that community feedback, with analytics on each model's strengths, weaknesses and tradeoffs, representative samples of the underlying feedback for auditability, and SLAs on delivery. At launch Arena cited more than 250 million conversations, over 2 million monthly votes and over 3 million monthly users. Investors and analysts describe private arenas for confidential pre-release or proprietary-data testing as part of the commercial offer. TechCrunch reported on October 8, 2026 that Arena raised a $200 million Series B at a $3.1 billion valuation, led by Lightspeed and Khosla Ventures, after a $150 million Series A in January 2026, and that annualized revenue reached $100 million in June 2026. The same week Arena added an alignment leaderboard that ranks models on unauthorized actions, false attribution and deceptive completion. The method has critics: an April 2025 paper, 'The Leaderboard Illusion,' found data access skewed toward large proprietary labs, and Surge AI argued in December 2025 that arena voting rewards style over correctness.
A head of AI or ML platform lead who has to pick between several frontier and open-weight models for a production use case and wants human preference data alongside internal evals.
Head-to-head human preference analytics on the candidate models, with auditable feedback samples and a committed delivery date.
At a Glance
- Category
- AI Models & APIs
- Pricing
- Freemium, Contact for pricing
- Target Market
- CTOs, Heads of AI, ML Engineers, Model Labs
- Deployment
- Cloud-only
- Founded
- 2025
- Headquarters
- San Francisco, United States
Key Features
- ✓Community-based model evaluations
Analyses drawn from millions of blind side-by-side human votes, showing how models perform on real prompts rather than fixed test sets.
- ✓Strength and tradeoff analytics
Reports break down where each model is strong or weak and what you trade off by picking one over another.
- ✓Auditable feedback samples
Customers get representative samples of the underlying human feedback, so the conclusions can be checked rather than taken on trust.
- ✓Delivery SLAs
Evaluation results come with committed delivery timelines, which matters when a model decision sits on a launch schedule.
- ✓Specialized arenas
Separate arenas for web development, search-augmented answers, vision and coding assistants let you compare models on the task type you actually ship.
- ✓Alignment leaderboard
A new category, added October 2026, ranks models on unauthorized actions, false attribution and claiming to finish tasks that were not done.
Use Cases
- •Model selection for an internal assistant
An enterprise compares three candidate LLMs on human preference data before standardizing one for an employee-facing chat assistant.
- •Pre-release testing at a model lab
A lab runs a confidential evaluation of an unreleased model against current leaders to decide whether it is ready to launch.
- •Agent safety review
A governance team checks the alignment leaderboard for deceptive-completion and unauthorized-action rankings before approving a model for agentic workflows.
- •Open-weight versus commercial tradeoff
A platform team weighs an open-weight model against a paid API using one shared methodology, to judge whether self-hosting costs quality.
Ideal For
Best For
- ✓Model labs benchmarking a pre-release model against competitors on real user prompts
- ✓Enterprises shortlisting an LLM for a chat, coding or search-augmented use case
- ✓AI governance teams that want an outside alignment signal (unauthorized actions, deceptive completion) next to internal red-teaming
- ✓Teams comparing open-weight and commercial models under one methodology
Not Ideal For
- ✗Teams that need correctness scored against ground truth on their own domain data: public votes reward tone and formatting, and critics including Surge AI showed confident wrong answers can win
- ✗Buyers who need published self-serve pricing or an instant trial, since the commercial service is sold through sales and Arena publishes no list prices
- ✗Organizations wary of a vendor that sells evaluation services to the same labs it ranks publicly
Deployment
Market Analysis
Pros
- ✓A large, continuously refreshed human preference dataset (3M+ monthly users, 250M+ conversations at the AI Evaluations launch)
- ✓Ranks commercial and open-weight models under one method, and keeps key datasets public on Hugging Face
- ✓Commercial traction: annualized revenue reportedly reached $100M in June 2026, under a year after launch
- ✓The new alignment leaderboard measures agent failure modes (deceptive completion, unauthorized action) that standard benchmarks miss
Cons
- ✗Votes can reward style over substance: Surge AI's December 2025 critique argued verbosity and formatting beat correctness, and that post drew 246 points on Hacker News
- ✗'The Leaderboard Illusion' (April 2025) found OpenAI and Google each received roughly a fifth of arena data and that private testing let big labs publish only their best variant
- ✗Meta's public Llama 4 differed from the variant that topped the leaderboard, which raised questions about gaming
- ✗Arena sells evaluations to the labs it ranks, a conflict of interest that analysts flag
- ✗No published pricing, and almost no detail on what an enterprise engagement includes
Pricing
Public arena and leaderboard
$0
- ✓Blind side-by-side model comparison
- ✓Public leaderboards across text, web development, search and vision
AI Evaluations
Contact for pricing
- ✓Community-based evaluation analytics
- ✓Auditable feedback samples
- ✓Delivery SLAs
The public arena and leaderboard are free. AI Evaluations is sold through sales with no published list price; Arena says it offers flexible pricing for open-weight model teams and nonprofits. Contrary Research reports Arena keeps about 80% of its preference data as a commercial asset.
Security & Compliance
Sources
This page was written from 7 sources, 6 on domains other than arena.ai.
- 1.arena.ai — ai evaluationsvendor
- 2.techcrunch.com — popular ai leaderboard arena nearly doubles valuation to 3 1
- 3.research.contrary.com — lmarena
- 4.testingcatalog.com — lmarena launches ai evaluations for enterprises and teams.md
- 5.a16z.com — investing in lmarena the reliability layer for ai
- 6.hn.algolia.com — search
- 7.plushcap.com — surge ai lmarena is a cancer on ai
Stay Ahead of the Curve
Weekly enterprise AI insights for technology leaders. No spam, no vendor pitches—unsubscribe anytime.
SubscribeRelated Products
Kimi K3
2.8T-parameter open-weight MoE model with a 1M-token context, priced at $3/$15 per million tokens
TypeSafe Jev
A 'System One' decision model that returns typed choices, scores and calibrated probabilities instead of text, for agent routing and classification
Xiaomi MiMo-V2.6
MIT-licensed, 1M-context multimodal MoE models that took the top open-weights score on Artificial Analysis at well under $1 per million tokens
PrismML Bonsai
Open-weight 1-bit and ternary LLMs that run 27B-class AI on laptops and phones