Launched May 2023 by LMSYS at UC Berkeley, now rebranded as LMArena. A crowdsourced evaluation platform for head-to-head model comparison. As of 2026, it has accumulated over 6 million votes across 100+ models, and is the most-cited LLM leaderboard in the world.
How it works
- User visits the site and types any prompt.
- Two randomly selected models respond side-by-side, identities hidden.
- User picks “A is better,” “B is better,” or “tie.”
- Model identities are revealed (this incentivizes participation — users want to find out).
- Vote is recorded.
What is Elo?
Elo comes from chess, invented by Arpad Elo. Players gain or lose rating points based on head-to-head outcomes. High-rated beats low-rated → small point change. Low-rated beats high-rated → large point change.
In the Arena, each pairwise vote is treated like a chess match. Over millions of votes, models converge to stable ratings reflecting relative strength.
The number is only meaningful relative to other models — 1,501 isn’t a percentage. A 100-point gap means the higher-rated model is preferred about 64% of the time (from the logistic function of the Elo difference).
Technically, the Arena uses the Bradley-Terry model rather than pure Elo. Both produce scalar ratings from pairwise comparisons; Bradley-Terry fits maximum likelihood across all comparisons simultaneously, while Elo updates incrementally. They converge to similar rankings.
The economics
Who pays for votes?
Nobody. It’s free crowdsourced labor. Users participate voluntarily — partly for curiosity, partly because the AI community wants better evaluation.
Who pays for compute?
Model providers themselves serve their models on the Arena. OpenAI, Anthropic, Google, Meta provide API access or hosting because a high Arena ranking has enormous marketing value. The cost of serving is trivial compared to the promotional value.
For open-weight models, LMSYS originally ran inference on donated GPU clusters from academic grants and sponsors. As the Arena grew, cloud providers and GPU companies took over much of the compute sponsorship.
LMSYS funding
Started as an academic project at UC Berkeley, funded by grants and compute donations. In 2024, LMSYS spun into a commercial entity (LMArena/OpenLM), raising concerns about independence — model providers are both sponsors and participants.
Strengths
- Captures real-world preferences on diverse, naturally occurring prompts.
- No fixed test set to contaminate or overfit.
- Rank correlation above 0.99 between monthly updates — stable.
Weaknesses and controversies
Verbosity / style bias. Users tend to prefer verbose, polished responses over concise correct ones. Models can game rankings by being more elaborate without being more helpful.
Sampling bias. Prompt distribution skews toward English and coding. Doesn’t represent global usage patterns.
The “Leaderboard Illusion.” Singh et al. (Cohere/AI2/Princeton/Stanford, 2025) documented that labs can submit many private variants and publicize only the best. Submitting 10 private variants inflates scores by roughly 100 Elo points.
The Llama 4 controversy (April 2025). Meta tested 27 private variants. The top scorer (1,417 Elo, 2nd place) was never publicly released. The shipped model performed at about 32nd–35th place. LMSYS acknowledged Meta’s approach “did not match what we expect from model providers.”
The structural conflict. The Arena’s value depends on being perceived as independent. But the model providers who fund it are the same ones being ranked. The Llama 4 incident made this conflict visible.
Does scoring high on Arena mean the model is good?
Scoring high means the model wins blind pairwise comparisons on Arena-style prompts. That’s not nothing. But as Llama 4 showed, a variant optimized for Arena vibes isn’t necessarily the best general-purpose model. If Meta didn’t ship the high-scoring variant, it implies that variant sacrificed something important — reliability, consistency, instruction-following — to win pairwise comparisons.
Goodhart’s Law at the product level: optimizing for the leaderboard metric produces a model that wins comparisons but isn’t the best model to deploy.