What is an AI benchmark?

An AI benchmark is a standard test for AI models. Everyone runs the same set of questions or tasks, and the results are scored the same way. That makes it possible to compare models fairly, the way a standardized exam lets you compare students from different schools.

What a benchmark actually measures

Each benchmark tests one kind of skill. Some examples you’ll see often:

General knowledge. Multiple-choice questions across many school and professional subjects. MMLU is a well-known example.
Hard reasoning. Questions written by experts so that the answer can’t simply be looked up. GPQA is one of these.
Coding. The model is asked to fix real problems in real software projects, and the fix is checked by running tests. SWE-bench works this way.
People’s preferences. Instead of a fixed answer key, people compare two anonymous answers side by side and vote for the better one. Arena (previously called LMArena) ranks models this way.

How to read a score

Most benchmark scores are a percentage: how many questions or tasks the model got right. A few things to keep in mind:

Compare like with like. A score only means something next to other scores on the same benchmark, run the same way.
Check the date. Models are updated often. A result from last year may not describe the model you can use today.
Small gaps may not matter. If two models are a point or two apart, you probably won’t notice a difference in everyday use.
Look for the method. A trustworthy result says which version of the model was tested, the settings used, and who ran the test.

Why a high score isn’t the whole story

Leaked test questions. If a benchmark’s questions end up in a model’s training data, the model can score well by remembering answers rather than reasoning. Researchers call this contamination.
Tests age. Once most top models score near the maximum on a test, it stops telling them apart, so newer, harder benchmarks replace it.
Your task is not the test. A model that is excellent at coding puzzles may be only average at writing a friendly email. The best check is to try a model on the kind of work you do.

The short version

Benchmarks are useful for narrowing down your options. They don’t replace trying a model yourself. Use them to build a shortlist, then test the shortlist on your own real tasks.