Building block 1: what is a benchmark, actually?
A benchmark is, at its core, nothing more than a questionnaire with known correct answers: a fixed set of test tasks where it's already clear what "correct" means. A model works through these tasks, and a scoring method counts how often its answer matches the known correct one. This simple principle is behind most "knowledge" benchmarks.
Building block 2: three fundamentally different ways of measuring
Not every benchmark measures the same way. Three types explain almost the entire landscape: with objectively correct benchmarks, there's a clearly right answer – a math problem, a multiple-choice question, whether a code fix passes the tests. With human preference, there's no "correct" answer; instead, people compare two answers and say which one they prefer. And with AI as judge, another language model scores the answer against a given rubric – used because it's far cheaper and faster than human evaluation.
Each of these three types has its own weaknesses, covered in detail in the following modules.
Building block 3: why sample size matters
A model answers 10 test questions, gets 8 right – "80%". But flip a fair coin 10 times, and you'll also land on 8 or more heads fairly often, purely by chance. Only with enough test questions – hundreds to thousands for real benchmarks – does a percentage become meaningful at all.
Building block 4: why a small difference is often not a real difference
71% vs. 73% sounds like a clear winner. But if the underlying sample is small or the task is hard, that gap can sit entirely within the range of chance. That's exactly what a confidence interval expresses, without needing to know the formula behind it: it says, in essence, "the true score is very likely somewhere in this range" – and if two models' ranges overlap, the claimed winner isn't actually established.
Why this matters to you as a decision-maker
These four building blocks are enough to critically read any benchmark number, even before knowing a specific benchmark in detail. The following modules look at the three measurement paradigms from building block 2 one at a time – and show where each of them runs into its limits in practice.