Benchmarks

Benchmarks

128 tracked evaluations · normalization bounds power the Index (see methodology)

Human preference

AlpacaEval[0–100] %

LLM-judged length-controlled win rate vs a reference model.

leaderTülu 2+DPO 70B95.1%

Arena-Hard v2[0–84] %

Second-generation Arena-Hard, harder prompt set.

leaderQwen3-Max86.1%

Arena-Hard[0–99] %

LLM-judged win rate on hard, real-world Arena-style prompts.

leaderQwen3-235B-A22B (Non-Thinking)96.1%

Arena Elo[1000–1460] Elo

Blind pairwise human votes (arena.ai). The gold standard for conversational preference.

leaderClaude Fable 51505

KoMT-Bench[3–9.5] /10

Korean-language multi-turn conversation quality, judged 1-10.

leaderLG EXAONE 3.0 7.8B Instruct8.92

LogicKor[3–9.5] /10

Korean-language logical/reasoning chat-quality judge benchmark.

leaderGPT-4o9.33

MT-Bench[4–9.5] /10

Multi-turn conversation quality, GPT-4-judged 1-10 scale.

leaderHunyuan-Large (A52B)9.4

OpenAssistant Bench[900–1320] Elo

Early open community Elo ranking, GPT-4-judged.

leaderGPT-41294

Rakuda[3–10] /10

Japanese-language absolute-quality chat evaluation, GPT-4-judged.

leaderPLaMo-100B9.73

Vicuna Bench[880–1200] Elo

Early human/LLM-judged Elo ranking on 80 diverse questions.

leaderGPT-41176

WildBench[0–70] %

LLM-judged win rate on real, in-the-wild user queries.

leaderMistral Small 3.2 (24B)65.3%

Knowledge

C-Eval[20–88] %

Comprehensive Chinese-language academic/professional knowledge exam.

leaderQwen3.6-Plus93.3%

CMMLU[20–87] %

Chinese-culture-aware multi-subject knowledge benchmark.

leaderDoubao-1.5-Pro90.9%

FACTS Grounding[40–90] %

Faithfulness of a model's response to a long provided context document.

leaderGemini 2.5 Flash-Lite (Thinking)86.8%

FIN-bench[20–69] %

Finnish-language knowledge and reasoning benchmark suite.

leaderPoro 34B66.3%

JASTER[20–81] %

Japanese-language academic/professional knowledge benchmark suite.

leaderPLaMo-100B77.5%

MMLU: Anatomy[40–86] %

Anatomy subject slice of MMLU.

leaderPalmyra Med 70B83.7%

MMLU: Clinical Knowledge[40–94] %

Clinical-knowledge subject slice of MMLU.

leaderPalmyra Med 70B90.9%

MMLU (EU-21 languages)[20–80] %

MMLU translated and averaged across 21 official EU languages.

leaderLlama 3.1 70B77.1%

MMLU: Medical Genetics[40–97] %

Medical-genetics subject slice of MMLU.

leaderPalmyra Med 70B94.0%

MMLU-Pro[15–91] %

Harder, 10-option MMLU successor with less shortcut-guessing.

leaderClaude Fable 591.5%

MMLU-Redux[40–96] %

Manually re-annotated, error-corrected subset of MMLU.

leaderQwen3.7-Max95.0%

MMLU (STEM)[30–83] %

STEM-subject slice of MMLU.

leaderFalcon-H1 34B83.6%

MMLU[40–97] %

57-subject multiple-choice knowledge. Largely saturated at the frontier.

leaderOpenAI o392.9%

MMMLU[30–88] %

Multilingual MMLU across 14 languages.

leaderGemini 3.1 Pro92.6%

PubMedQA[50–80] %

Biomedical-literature yes/no/maybe question answering.

leaderPalmyra Med 70B79.6%

SimpleQA[0–66] %

Short factual questions with a single verifiable answer; notoriously low scores.

leaderGPT-4.562.5%

SuperCLUE[40–90] %

Comprehensive Chinese-language LLM evaluation (CLUEbenchmark), covering reasoning, knowledge, and generation.

leaderBaichuan 480.6%

TriviaQA[30–94] %

Closed-book trivia question answering.

leaderSarvam-1 (2B)90.6%

TruthfulQA[20–80] %

Questions designed to elicit common human misconceptions.

leaderPhi-3.5-MoE (16x3.8B, 6.6B active)77.5%

Reasoning

AGIEval[20–92] %

Human standardized exams (SAT/LSAT/Gaokao) repurposed as an LLM benchmark.

leaderOLMo 3-Think 32B88.2%

ANLI[30–62] %

Adversarial natural-language-inference benchmark.

leaderPhi-3-small (7B)58.1%

ARC-AGI-1[0–91] %

Abstraction-and-reasoning visual puzzle grid tasks, v1.

leaderGPT-5.696.5%

ARC-AGI-2[0–95] %

Harder abstraction-and-reasoning puzzle grid tasks, v2.

leaderGPT-5.692.5%

ARC-Challenge[25–100] %

Grade-school science questions, challenge split.

leaderLlama 3.1 405B96.9%

ARC-Easy[40–99] %

Grade-school science questions, easy split.

leaderPhi-3-medium (14B)97.7%

BIG-Bench Hard[30–97] %

23 of the hardest BIG-bench tasks, chain-of-thought.

leaderERNIE 4.5 300B-A47B94.3%

BIG-bench[20–68] %

Broad collaborative benchmark of 200+ diverse reasoning tasks.

leaderChinchilla65.1%

DROP[20–92] F1

Discrete reasoning over paragraphs requiring numerical/set operations.

leaderHunyuan-T193.1

GPQA Diamond[20–99] %

Graduate-level science questions that resist memorization.

leaderGPT-5.694.6%

HellaSwag[25–100] %

Commonsense sentence-completion, adversarially filtered.

leaderClaude 3 Opus95.4%

Humanity's Last Exam[0–65] %

2,500 expert-written frontier questions across 100+ subjects.

leaderClaude Sonnet 557.4%

IFBench[20–86] %

Extended verifiable instruction-following benchmark.

leaderMiniMax M383.0%

IFEval[20–96] %

Verifiable instruction-following (format/length/content constraints).

leaderGemma 4 26B A4B98.5%

LAMBADA[40–89] %

Broad-context last-word prediction requiring long-range coherence.

leaderGPT-3 175B86.4%

OpenBookQA[25–93] %

Elementary science QA requiring open-book fact combination.

leaderClaude 190.8%

PIQA[50–92] %

Physical-interaction commonsense QA.

leaderGPT-4o mini93.1%

RACE-H[30–96] %

High-school-level reading-comprehension multiple choice.

leaderClaude 3 Opus92.9%

Social IQa[40–84] %

Social/emotional commonsense reasoning about everyday situations.

leaderApple DCLM-Baseline 7B82.9%

StoryCloze[50–91] %

Commonsense story-ending selection.

leaderGPT-3 175B87.7%

SuperGPQA[10–86] %

Expanded graduate-level science reasoning across 285 disciplines.

leaderQwen3.7-Max73.6%

SuperGLUE[50–93] %

Harder successor to GLUE spanning 8 diverse language-understanding tasks.

leaderUL2 20B90.7%

WinoGrande[50–92] %

Large-scale Winograd-schema commonsense pronoun resolution.

leaderPaLM 290.9%

XWinograd[50–88] %

Cross-lingual Winograd-schema commonsense resolution.

leaderSarashina2-70B91.8%

Coding

Aider Polyglot[0–93] %

Real-world multi-language code-editing benchmark used by the Aider tool.

leaderClaude Opus 4.589.4%

BigCodeBench[0–88] %

Practical, tool-using code generation with complex instructions.

leaderGPT-4o mini57.4%

BIRD-SQL[0–58] %

Text-to-SQL generation over large, real-world databases.

leaderCommand A59.5%

Codeforces[400–3300] Elo

Competitive-programming rating estimated from Codeforces-style problems.

leaderDeepSeek-V4-Pro (Think Max)3206

CyberGym[0–87] %

Real-world cybersecurity vulnerability discovery and exploitation tasks.

leaderGPT-5.684.5%

DS-1000[0–36] %

Realistic data-science coding problems across 7 Python libraries.

leaderStarCoder2 15B33.8%

HumanEval FIM[0–91] %

Fill-in-the-middle code completion variant of HumanEval.

leaderCodestral 22B91.6%

HumanEval+[10–100] %

EvalPlus-augmented HumanEval with extra test cases (stricter than base HumanEval).

leaderMistral Small 3.2 (24B)92.9%

HumanEval[10–100] %

Hand-written Python programming problems, pass@1. Long-saturated at the frontier.

leaderClaude Opus 4.599.4%

LiveCodeBench Pro[1000–2990] Elo

Elo-rated variant of LiveCodeBench judged against competitive-programmer ratings.

leaderGemini 3.1 Pro2887

LiveCodeBench[0–94] %

Contamination-free competitive programming, rolling problem set.

leaderDeepSeek-V4-Pro (Think Max)93.5%

MBPP+[10–90] %

EvalPlus-augmented MBPP with extra test cases (stricter than base MBPP).

leaderLlama 3.1 405B88.6%

MBPP[10–96] %

Mostly Basic Python Problems, pass@1.

leaderLlama-3.3-Nemotron-Super-49B v1 (Reasoning On)91.3%

RepoBench[0–40] %

Repository-level code auto-completion benchmark.

leaderDeepSeek Coder 33B41.9%

SWE-bench Pro[0–85] %

Harder, held-out variant of SWE-bench Verified.

leaderClaude Fable 580.0%

SWE-bench Verified[0–97] %

Resolving real GitHub issues in real repositories, verified subset.

leaderClaude Fable 595.0%

Terminal-Bench 2.0[0–89] %

Second-generation Terminal-Bench, harder task suite.

leaderGPT-5.688.8%

Terminal-Bench[0–94] %

Autonomous shell/terminal task completion in realistic environments.

leaderClaude Opus 4.5 (High)59.3%

Math

AIME[0–100] %

American Invitational Mathematics Examination, pass@1.

leaderGPT-5.2100.0%

American Mathematics Competitions problem set.

leaderFalcon-H1 34B69.4%

FrontierMath (Tiers 1-3)[0–94] %

Novel unpublished research-level math problems, easier tiers.

leaderGPT-5.689.0%

FrontierMath (Tier 4)[0–88] %

FrontierMath hardest tier — IMO/research-adjacent difficulty.

leaderGPT-5.683.0%

Gaokao Math[0–100] %

China's national college entrance exam, mathematics paper.

leaderDoubao-Seed-1.696.0%

GSM8K[5–100] %

Grade-school math word problems, 8-shot pass@1. Saturated at the frontier.

leaderLlama 3.1 405B96.8%

MATH-500[20–100] %

500 competition math problems across difficulty levels.

leaderGPT-599.4%

MGSM[10–95] %

Multilingual grade-school math word problems.

leaderOpenAI o4-mini93.7%

MiniF2F (test)[0–94] %

Formal (Lean 4) theorem-proving benchmark, test split.

leaderDeepSeek-Prover-V2 671B88.9%

MiniF2F (valid)[0–96] %

Formal (Lean 4) theorem-proving benchmark, validation split.

leaderDeepSeek-Prover-V2 671B90.6%

OlympiadBench[0–65] %

Olympiad-level bilingual multimodal science/math problems.

leaderDoubao-1.5-Pro59.8%

OmniMath[0–81] %

Broad olympiad-level mathematics problem set.

leaderPhi-4-reasoning76.6%

PutnamBench[0–8] %

Formal (Lean 4) proofs of William Lowell Putnam Competition problems.

leaderDeepSeek-Prover-V2 671B7.1%

USAMO[0–90] %

USA Mathematical Olympiad problem set.

leaderClaude Opus 4.896.7%

Vision

AI2D[40–97] %

Diagram question answering (science-textbook diagrams).

leaderMolmo 72B96.3%

ChartQA[30–95] %

Question answering over charts and graphs.

leaderMiniMax-VL-0191.7%

CharXiv[10–89] %

Chart understanding drawn from real arXiv papers.

leaderMuse Spark 1.1 (xhigh)88.4%

COCO Captions[0–155] CIDEr

Zero-shot COCO image-captioning quality (CIDEr).

leaderCogVLM-17B148.7

DocVQA[50–99] %

Question answering over scanned document images.

leaderQwen2-VL-72B96.5%

Flickr30k[0–91] CIDEr

Zero-shot image-captioning quality (CIDEr).

leaderCogVLM-17B94.9

Compositional visual reasoning question answering.

leaderQVQ-72B-Preview63.9%

MathVision[0–92] %

Harder visual math-competition problem set.

leaderSeed 2.1 Pro92.6%

MathVista[10–96] %

Visual mathematical reasoning over charts, diagrams and figures.

leaderSeed 2.1 Pro90.7%

MMBench (Chinese)[40–94] %

Chinese split of MMBench.

leaderERNIE 4.5 VL 424B-A47B90.9%

MMBench (English)[40–96] %

English split of MMBench.

leaderQwen3.5-397B-A17B93.7%

MMBench[40–85] %

Broad multimodal ability benchmark with circular-eval robustness check.

leaderPhi-3.5-vision (4.2B)81.9%

MMMU-Pro[15–88] %

Harder, vision-only-input variant of MMMU.

leaderClaude Opus 4.785.5%

MMMU[20–88] %

College-level multimodal understanding across 30 disciplines.

leaderClaude Fable 589.3%

OCRBench[300–938] /1000

Text-recognition-in-images benchmark across 5 OCR task types.

leaderInternVL3-78B906

OK-VQA[30–68] %

VQA requiring outside/commonsense knowledge beyond the image.

leaderCogVLM-17B64.8%

RefCOCO+[50–94] %

RefCOCO variant excluding absolute-location expressions.

leaderDeepSeek-VL291.2%

RefCOCO[50–92] %

Referring-expression visual grounding accuracy.

leaderQwen3.5-Omni-Plus95.0%

SEED-Bench[40–82] %

Broad generative multimodal comprehension benchmark.

leaderNAVER HyperCLOVA X SEED Think 32B77.9%

StreamingBench[30–68] %

Real-time, live-streaming video comprehension benchmark.

leaderGemini 1.5 Pro70.3%

TextVQA[28–90] %

VQA requiring reading text embedded in images.

leaderMolmo 2 8B85.7%

Video-MME[30–90] %

Broad video understanding across short/medium/long clips.

leaderSeed 2.1 Pro89.2%

VQAv2[40–87] %

Open-ended visual question answering.

leaderReka Edge (2026)88.4%

Agents

Agents' Last Exam[0–56] %

Frontier agentic-capability exam across long-horizon tool-use tasks.

leaderGPT-5.652.7%

BFCL v3[20–82] %

Third-generation Berkeley Function-Calling Leaderboard.

leaderHunyuan-A13B78.3%

BFCL[20–89] %

Berkeley Function-Calling Leaderboard, tool/API call accuracy.

leaderSmolLM3 3B (No Thinking)92.3%

BrowseComp[0–96] %

Hard-to-find-information web-browsing agent benchmark.

leaderKimi K391.2%

General-assistant tasks requiring reasoning, tool use and web browsing.

leaderMiniMax-M275.7%

GDPval-AA[0–2000] pts

Updated/harder economically-valuable-work evaluation (0-2000 scale) — distinct from the original GDPval benchmark.

leaderClaude Fable 51932

GDPval[0–90] %

Economically valuable real-world professional task completion, wins-or-ties.

leaderSeed 2.1 Pro87.9%

APIBench (HuggingFace)[0–76] %

Zero-shot, no-retriever API-call generation over HuggingFace APIs.

leaderGorilla 7B71.7%

MCP Atlas[50–90] %

Multi-server Model Context Protocol tool-use benchmark across realistic agentic workflows.

leaderKimi K384.2%

OSWorld-Verified[0–90] %

Verified/cleaned subset of OSWorld — real desktop-GUI computer-use tasks scored pass@1.

leaderClaude Fable 585.0%

OSWorld[0–88] %

Original OSWorld computer-use benchmark (screenshot-only agent configuration) — distinct from the later OSWorld-Verified subset.

leaderSeed 2.1 Pro78.8%

PinchBench[0–96] %

Agent productivity benchmark for long-running task completion.

leaderTrinity-Large-Thinking91.9%

tau-bench[0–100] %

Tool-agent-user interactions in realistic airline/retail domains.

leaderClaude Opus 4.182.4%

τ²-Bench Airline[20–95] %

τ²-Bench airline domain — dual-control tool-agent-user tasks in a simulated airline support setting; a harder successor to tau-bench, distinct scale.

leaderTrinity-Large-Thinking88.0%

τ²-Bench Retail[30–99] %

τ²-Bench retail domain — dual-control tool-agent-user tasks in a simulated e-commerce/retail support setting; a harder successor to tau-bench, distinct scale.

leaderClaude Sonnet 4.691.7%

τ²-Bench Telecom[30–99] %

τ²-Bench telecom domain — dual-control tool-agent-user tasks in a simulated telecom support setting; a harder successor to tau-bench, distinct scale.

leaderClaude Opus 4.699.3%

τ³-Banking[0–45] %

τ³-Bench banking domain (Artificial Analysis) — a newer, substantially harder generation of dual-control tool-agent-user tasks than τ²-Bench; scores are much lower across the field.

leaderGrok 4.533.0%

APIBench (TensorHub)[0–89] %

Zero-shot, no-retriever API-call generation over TensorHub APIs.

leaderGorilla 7B83.8%

APIBench (TorchHub)[0–63] %

Zero-shot, no-retriever API-call generation over TorchHub APIs.

leaderGorilla 7B59.1%