Benchmarks

LLM-judged length-controlled win rate vs a reference model.

Second-generation Arena-Hard, harder prompt set.

LLM-judged win rate on hard, real-world Arena-style prompts.

Blind pairwise human votes (arena.ai). The gold standard for conversational preference.

Korean-language multi-turn conversation quality, judged 1-10.

Korean-language logical/reasoning chat-quality judge benchmark.

Multi-turn conversation quality, GPT-4-judged 1-10 scale.

Early open community Elo ranking, GPT-4-judged.

Japanese-language absolute-quality chat evaluation, GPT-4-judged.

Early human/LLM-judged Elo ranking on 80 diverse questions.

LLM-judged win rate on real, in-the-wild user queries.

Comprehensive Chinese-language academic/professional knowledge exam.

Chinese-culture-aware multi-subject knowledge benchmark.

Faithfulness of a model's response to a long provided context document.

Finnish-language knowledge and reasoning benchmark suite.

HLE-Verified — the verified subset of Humanity's Last Exam, filtered for unambiguous ground truth. Scores are not comparable with the full HLE set filed under `hle`.

Japanese-language academic/professional knowledge benchmark suite.

Anatomy subject slice of MMLU.

Clinical-knowledge subject slice of MMLU.

MMLU translated and averaged across 21 official EU languages.

Medical-genetics subject slice of MMLU.

Harder, 10-option MMLU successor with less shortcut-guessing.

Manually re-annotated, error-corrected subset of MMLU.

STEM-subject slice of MMLU.

57-subject multiple-choice knowledge. Largely saturated at the frontier.

Multilingual MMLU across 14 languages.

Biomedical-literature yes/no/maybe question answering.

Short factual questions with a single verifiable answer; notoriously low scores.

Comprehensive Chinese-language LLM evaluation (CLUEbenchmark), covering reasoning, knowledge, and generation.

Closed-book trivia question answering.

Questions designed to elicit common human misconceptions.

Human standardized exams (SAT/LSAT/Gaokao) repurposed as an LLM benchmark.

Adversarial natural-language-inference benchmark.

Abstraction-and-reasoning visual puzzle grid tasks, v1.

Harder abstraction-and-reasoning puzzle grid tasks, v2.

Interactive abstraction-and-reasoning benchmark (ARC Prize): agents explore novel turn-based environments with no instructions, infer goals on the fly, build world models, and adapt across increasingly hard levels. v3.

Grade-school science questions, challenge split.

Grade-school science questions, easy split.

23 of the hardest BIG-bench tasks, chain-of-thought.

Broad collaborative benchmark of 200+ diverse reasoning tasks.

Discrete reasoning over paragraphs requiring numerical/set operations.

Graduate-level science questions that resist memorization.

Commonsense sentence-completion, adversarially filtered.

2,500 expert-written frontier questions across 100+ subjects.

Extended verifiable instruction-following benchmark.

Verifiable instruction-following (format/length/content constraints).

Broad-context last-word prediction requiring long-range coherence.

OpenAI MRCR v2, 8-needle variant, 512K-1M token band — the harder long-context retrieval band, where scores separated sharply through 2026.

OpenAI MRCR v2, 8-needle variant, 256K-512K token band — pure long-context retrieval: reproduce a target turn from among 8 needles spread across the context. Graded by sequence-matcher ratio.

Elementary science QA requiring open-book fact combination.

Physical-interaction commonsense QA.

High-school-level reading-comprehension multiple choice.

Social/emotional commonsense reasoning about everyday situations.

Commonsense story-ending selection.

Expanded graduate-level science reasoning across 285 disciplines.

Harder successor to GLUE spanning 8 diverse language-understanding tasks.

Large-scale Winograd-schema commonsense pronoun resolution.

Cross-lingual Winograd-schema commonsense resolution.

Real-world multi-language code-editing benchmark used by the Aider tool.

Practical, tool-using code generation with complex instructions.

Text-to-SQL generation over large, real-world databases.

Competitive-programming rating estimated from Codeforces-style problems.

CursorBench 3.2 — Cursor's independently-run agentic coding evaluation over real editor sessions. Reported by vendors from Cursor's runs rather than self-administered.

CWE-Bench, run by Collinear — an external benchmark for automated vulnerability patching across CWE classes. Reported as pass@1.

Real-world cybersecurity vulnerability discovery and exploitation tasks.

DeepSWE v1.1 — 113 long-horizon software-engineering tasks written from scratch to avoid contamination, measuring frontier coding agents on diverse real-world complexity. Mean over 5 trials.

Realistic data-science coding problems across 7 Python libraries.

FrontierCode v1.1 Extended set — Cognition's agentic coding benchmark built from real open-source pull requests, scored on the wider Extended split rather than the 150-task Main set. Not comparable with the Main-set scores filed under `frontiercode`.

FrontierCode v1.1 (Main set) — 150 agentic software-engineering tasks derived from real open-source pull requests, run and graded by Cognition against held-out unit tests plus weighted code-quality rubrics. Composite score, mean@5.

Fill-in-the-middle code completion variant of HumanEval.

EvalPlus-augmented HumanEval with extra test cases (stricter than base HumanEval).

Hand-written Python programming problems, pass@1. Long-saturated at the frontier.

Elo-rated variant of LiveCodeBench judged against competitive-programmer ratings.

Contamination-free competitive programming, rolling problem set.

EvalPlus-augmented MBPP with extra test cases (stricter than base MBPP).

Mostly Basic Python Problems, pass@1.

Generating a complete, working code repository from a natural-language specification.

Repository-level code auto-completion benchmark.

SciCode — research-grade scientific coding problems drawn from published computational-science workflows, decomposed into subproblems a model must implement and pass.

SRE-Bench — software reverse-engineering tasks: recover a binary's core logic without access to source. Reported pass@1 unless a row states an attempt budget.

SWE-Atlas Codebase QnA — 124 tasks over 11 real repositories testing deep code comprehension and question answering rather than patch generation. Mean pass@1 under a rubric judge.

SWE-bench Multilingual — 300 real-issue problems spanning 9 programming languages, extending SWE-bench beyond Python.

SWE-bench Multimodal — issues augmented with visual context (screenshots, design mockups), testing whether coding agents generalize to visual software domains.

Harder, held-out variant of SWE-bench Verified.

Resolving real GitHub issues in real repositories, verified subset.

Second-generation Terminal-Bench, harder task suite.

Autonomous shell/terminal task completion in realistic environments.

American Invitational Mathematics Examination, pass@1.

American Mathematics Competitions problem set.

Novel unpublished research-level math problems, easier tiers.

FrontierMath hardest tier — IMO/research-adjacent difficulty.

China's national college entrance exam, mathematics paper.

Grade-school math word problems, 8-shot pass@1. Saturated at the frontier.

500 competition math problems across difficulty levels.

Multilingual grade-school math word problems.

Formal (Lean 4) theorem-proving benchmark, test split.

Formal (Lean 4) theorem-proving benchmark, validation split.

Olympiad-level bilingual multimodal science/math problems.

Broad olympiad-level mathematics problem set.

Formal (Lean 4) proofs of William Lowell Putnam Competition problems.

USA Mathematical Olympiad problem set.

Diagram question answering (science-textbook diagrams).

BenchCAD Vision2Code — reconstruct a CAD program from rendered views of a part; scored by geometric overlap of the resulting 3D model against the reference.

Question answering over charts and graphs.

Chart understanding drawn from real arXiv papers.

Zero-shot COCO image-captioning quality (CIDEr).

Question answering over scanned document images.

Zero-shot image-captioning quality (CIDEr).

GDP.PDF — expert comprehension of complex real-world PDFs (tables, charts, dense multi-column layout), a document-understanding companion to GDPval.

Compositional visual reasoning question answering.

LVBench — long-video understanding over hour-scale footage, testing retrieval and reasoning across extended temporal context.

Harder visual math-competition problem set.

Visual mathematical reasoning over charts, diagrams and figures.

Chinese split of MMBench.

English split of MMBench.

Broad multimodal ability benchmark with circular-eval robustness check.

Harder, vision-only-input variant of MMMU.

College-level multimodal understanding across 30 disciplines.

Text-recognition-in-images benchmark across 5 OCR task types.

VQA requiring outside/commonsense knowledge beyond the image.

RefCOCO variant excluding absolute-location expressions.

Referring-expression visual grounding accuracy.

ScreenSpot-Pro — GUI element grounding on high-resolution professional-application screenshots: locate the described control precisely enough to click it.

Broad generative multimodal comprehension benchmark.

Real-time, live-streaming video comprehension benchmark.

VQA requiring reading text embedded in images.

Broad video understanding across short/medium/long clips.

Open-ended visual question answering.

Artificial Analysis Briefcase — an Elo-rated evaluation of real professional knowledge work (documents, analysis, deliverables), reported on a ~0-2000 scale. Companion to GDPval-AA.

Frontier agentic-capability exam across long-horizon tool-use tasks.

AndroidWorld — mobile computer-use benchmark: complete real tasks in Android apps through the device UI.

Zapier's cross-application workflow-orchestration benchmark: whether an agent can complete real business tasks end to end via REST/tool calls. Scored on a held-out private set where every assertion must pass.

Third-generation Berkeley Function-Calling Leaderboard.

Berkeley Function-Calling Leaderboard, tool/API call accuracy.

Hard-to-find-information web-browsing agent benchmark.

DeepSearchQA — 900 agentic browsing questions whose answers are lists of items; the agent researches each with search/open/find tools and is graded by semantic set matching (F1).

ExploitBench — turns known, disclosed vulnerabilities into working exploits end to end, the step above vulnerability discovery on the exploitation chain. Public-set scores are contamination-sensitive: labs increasingly report private internal ports alongside them.

Frontier agent-work benchmark from the Terminal-Bench/Harbor team (formerly Terminal-Bench 3.0): 74 tasks across 7 domains in v0.1, spanning artifacts from databases and ML checkpoints to CAD files and formal proofs; resolution rate per agent-harness x model pair, with per-trial token and cost accounting.

General-assistant tasks requiring reasoning, tool use and web browsing.

GAIA2 — second-generation general AI assistant benchmark: multi-step, tool-using real-world assistant tasks with verifiable answers.

Updated/harder economically-valuable-work evaluation (0-2000 scale) — distinct from the original GDPval benchmark.

Economically valuable real-world professional task completion, wins-or-ties.

Harvey's Legal Agent Benchmark (Vals AI) — complex, multi-step legal workflows scored by practising lawyers. Scores stay low in absolute terms: the frontier is in the teens.

Zero-shot, no-retriever API-call generation over HuggingFace APIs.

JobBench — 65 professional tasks across 35 white-collar occupations, each asking the agent to produce a practical work product scored against task-specific rubrics. Mean rubric score.

Multi-server Model Context Protocol tool-use benchmark across realistic agentic workflows.

OSWorld 2.0 — a harder, refreshed version of the OSWorld real-desktop-GUI computer-use benchmark, scored pass@1. Distinct from and not comparable to OSWorld-Verified (v1).

Verified/cleaned subset of OSWorld — real desktop-GUI computer-use tasks scored pass@1.

Original OSWorld computer-use benchmark (screenshot-only agent configuration) — distinct from the later OSWorld-Verified subset.

Agent productivity benchmark for long-running task completion.

Tool-agent-user interactions in realistic airline/retail domains.

τ²-Bench airline domain — dual-control tool-agent-user tasks in a simulated airline support setting; a harder successor to tau-bench, distinct scale.

τ²-Bench retail domain — dual-control tool-agent-user tasks in a simulated e-commerce/retail support setting; a harder successor to tau-bench, distinct scale.

τ²-Bench telecom domain — dual-control tool-agent-user tasks in a simulated telecom support setting; a harder successor to tau-bench, distinct scale.

τ³-Bench banking domain (Artificial Analysis) — a newer, substantially harder generation of dual-control tool-agent-user tasks than τ²-Bench; scores are much lower across the field.

Zero-shot, no-retriever API-call generation over TensorHub APIs.

Terminal-Bench 3.0 — the third-generation terminal-agent suite, a large step up in difficulty from 2.1 (frontier models score in the 20-35% range where they clear 85% on 2.1). General agent capability in a real shell, not coding alone.

Terminal-Bench 4.0 — 66-task successor suite spanning computational biology, physics simulation, CAD and formal proofs, run in a real terminal environment. Measures general long-horizon agent capability rather than coding specifically.

Terminal-Bench-Science 0.1 — a Stanford-led set of 70 agentic scientific workflows (life, physical, earth, mathematical and engineering sciences) executed in a terminal: read data, run computations, submit results. Standard error is roughly +/-3.5-4.5 points per model.

Toolathlon Verified — multi-tool agentic task suite over realistic tool ecosystems, verified split.

Zero-shot, no-retriever API-call generation over TorchHub APIs.

Vals Finance Agent v2 — Vals AI's independently-run benchmark of financial-analyst agent workflows over real filings and market documents.

WebArena-Verified — the human-verified split of WebArena, measuring autonomous browser use across realistic self-hosted web applications.