How to read AI benchmarks

AI companies and researchers love to publish scores. This guide explains, in plain language, what those numbers mean and how much weight to give them. New to the idea? Start with What is an AI benchmark?

The three kinds of score you’ll see most

1. Percent correct

Most benchmarks are a fixed set of questions with known answers. The score is simply the share the model got right. For example, MMLU asks multiple-choice questions across dozens of school and professional subjects (the MMLU paper).

How to read it: compare models on the same benchmark only, and remember that a multiple-choice test with four options gives about 25% just by guessing.

2. Elo or arena ratings

Some leaderboards don’t use an answer key. Instead, people compare two anonymous answers side by side and pick the better one, and those votes are turned into a rating, like chess Elo ratings (how Chatbot Arena works).

How to read it: a rating measures which answers people preferred, not whether they were correct. Ratings only mean something relative to other models on the same leaderboard at the same time, and small gaps are often within the margin of error.

3. pass@1 (for coding)

Coding benchmarks often run the model’s code against tests. pass@1 is the share of problems the model solved on its first attempt (the paper that introduced pass@k). A “pass@10” score, if you see one, lets the model try ten times, so it will always look higher.

Which benchmarks matter for you

For everyday use (writing, summarizing, explaining, answering questions):

• Broad knowledge and reasoning tests, and human-preference ratings, are the most relevant.
• Combined indexes help too. Epoch AI publishes the Epoch Capabilities Index, which combines many benchmarks into one number.
• Most importantly: try the model on your own tasks. See How to pick an AI model.

Specialist benchmarks (only matter if you do this work):

• Advanced math, like FrontierMath from Epoch AI.
• Software engineering, like SWE-bench, which uses real GitHub issues.
• Graduate-level science questions, like GPQA.

A model that tops a math leaderboard isn’t necessarily better at writing your emails.

Caveats to keep in mind

• Contamination. If test questions leaked into a model’s training data, it may have memorized answers. That’s called contamination, and it makes scores look better than real ability.
• Self-reported scores. Numbers in a company’s own announcement were run by that company, with settings it chose. They can be accurate, but they’re not independent. See self-reported score.
• Saturation. When top models all score near the maximum, a benchmark stops telling them apart. Epoch AI, for example, describes the MATH Level 5 benchmark as reaching saturation, which is why harder tests keep appearing.
• Cost and speed aren’t in the score. A benchmark doesn’t tell you what a model costs or how fast it answers. For that, look at model metadata, such as the prices and context window sizes listed on OpenRouter’s models page. Prices are usually per million tokens and change often.
• Sources differ. Different groups run the same benchmark with different prompts, settings, and versions, so the “same” score can differ between sites. Compare numbers from one source at a time, and check its date.

A quick checklist

1. Who ran the test: the model maker or an independent group?
2. When? Scores go stale quickly.
3. Is it the same benchmark and version as the number you’re comparing it to?
4. Does the benchmark match what you actually want to do?
5. What will it cost, and how fast is it, for your use?

For how Fanetti AI chooses and presents data, see our methodology.

Sources and credits: benchmark descriptions from Epoch AI, whose data is published under the Creative Commons Attribution (CC BY) license. Model price and context-window information is from OpenRouter. Both checked October 22, 2026. This guide doesn’t reproduce leaderboard numbers; follow the links for current results.