Why static benchmarks don't capture agentic behavior
A classic benchmark checks a single answer to a single question. An AI agent, by contrast, plans multiple steps, calls tools, checks their outputs, revises its approach, and has to keep going even after a setback. Whether the final answer is correct says little about whether the agent got lost along the way, picked the wrong tool and recovered, or got stuck in an unnoticed loop.
METR's time horizon: a new approach
The research organization METR instead measures how long a task would take a human that a model can still solve autonomously at a given success rate – a model's "time horizon". The numbers are remarkably concrete: GPT-2 sat at about 2 seconds, Claude 3.7 Sonnet at around 50 minutes, o3 at nearly 2 hours, and Opus 4.6 already reached about 12 hours (at a 50% success rate). For roughly six years, this time horizon has doubled on average about every seven months.
The portfolio approach instead of a single benchmark
Every module in this course points to the same recommendation: no single benchmark type is enough. A solid comparison triangulates several sources – an objectively-correct benchmark that fits the task (module 3), a human-preference ranking read alongside its confidence intervals (module 4), an agentic benchmark or time horizon score where relevant, and your own current test with real requests from your own use case – precisely because a public benchmark can be contaminated or saturated (modules 3 and 6).
How Beyond Prompt applies this itself
This website's own comparison section (see "Methodology" at /vergleiche/methodik) follows the same principle: performance criteria rest, where possible, on independent sources like Artificial Analysis or SWE-bench rather than pure vendor claims – and are checked regularly against their source, so outdated values don't quietly stay in place.
Why this matters to you as a decision-maker
The five steps in the interaction below sum up this entire course into a repeatable process you can apply to any future vendor claim – no matter which model is being marketed as "the best" at the moment.