Methodology

The Elo rating

Models are ranked by direct comparison, not by averaging incomparable score scales. For every benchmark, each pair of models that both report a headline score becomes one head-to-head battle: the higher raw score wins, an exactly equal score is a draw. Because both scores come from the same benchmark, no cross-benchmark normalization is needed — the comparison is apples to apples by construction. A Bradley-Terry model (the same statistical machinery behind LMArena's leaderboard) is then fitted over all battles, propagating strength through shared opponents, so two models are comparable even when they were never measured on the same benchmark.

P(A beats B) = s(A) / (s(A) + s(B))     — Bradley-Terry win model
rating       = 400·log10(s) + 1000       — Elo scale: +400 ⇒ 10:1 odds

battles: every benchmark two models both report = 1 head-to-head
fit:     maximum likelihood (MM algorithm), ties count as half-wins
anchor:  every model gets one pseudo-draw vs a fixed 1000-rated anchor,
         so undefeated models stay finite and the scale stays pinned

The rating is jointly fitted, so adding results moves every number slightly — ratings are published to 0.1 and each release is a reviewable diff. A 400-point gap means 10:1 expected win odds on a shared benchmark; frontier models rate near 3000 while early-generation models can rate below zero. The one Elo-unit benchmark in the catalog (Arena) participates like any other: only the ordering of two models' scores on it matters. Category indexes and the compare radar still use per-category normalized means against curated bounds — they are a capability profile, not a ranking.

Ranking eligibility

A rating is computed for every model, but a model earns an overall rank only once it has been evaluated on enough of the field to compare fairly — at least three benchmarks spanning at least two categories. Otherwise a model with a single cherry-picked high score would outrank a broadly-benchmarked frontier model. Below that floor a model is shown unrated (its rating still displayed for reference) and sorted after every ranked model, never erased.

Provenance

Every stored result carries a source: independent, arena, admin-run, curated, or self-reported. When a model×benchmark has rows from several sources, the headline score is picked in that order of precedence — independent measurements always beat a vendor's own numbers. The current dataset (537 models, as of July 2026) draws on a mix of self-reported, independent, and arena sources — each stored and displayed separately, never collapsed into a single number. Leaderboards show a per-row provenance badge so this is never hidden.

Movers & lineage

Version lineage links each model to its nearest strictly-older family member; same-day releases are size variants, not successions, and have no predecessor. “Biggest movers” are the largest positive Elo gains across those lineage edges.

Hardware fit

Fit verdicts use curated Q4 VRAM figures (ground truth beats formulas), with a 1.08× overhead factor for KV-cache and runtime. Mac budgets are already unified-memory discounted in the curated data. Where no curated figure exists the estimate is params × 4.5/8 × 1.08.

required = vramQ4 × 1.08        ratio = required / budget
ratio ≤ 0.8   fits comfortably
ratio ≤ 1.0   fits (tight)        ← the boolean "fits" everywhere else
ratio ≤ 1.3   partial offload
otherwise     won't run

tok/s is shown only where a measured throughput row exists — never interpolated.
MoE models need total-parameter memory; speed tracks active parameters.

Versioned snapshots

Data is curated in git, validated, and built into an immutable snapshot bundled with each deploy (currently v4168328496). The version is a content hash of the data, so it changes only when the data does — it keys every cache, so nothing is ever purged and the served numbers are always reproducible from the commit that shipped them.

Freshness & contributions

Curation is deliberate, not automated — which means the catalog can lag frontier releases by days. The dataset is a reviewable set of JSON/CSV files; corrections and additions arrive as pull requests where CI enforces referential integrity, score bounds and provenance before anything ships. See CONTRIBUTING in the repository, or start from the benchmark definitions.