Reading Benchmark Cards Critically

Every frontier model launches with a benchmark card — a wall of percentages on standard tests. Treat the card as marketing first, evidence second. The same number often hides different methodologies, contamination risks, and prompt engineering tricks.

Common benchmarks and what they measure

Three questions to ask any benchmark card

1. 1Was the test set in the training data — explicitly or by leak?
2. 2What inference scaffolding was used — chain-of-thought, multi-sample voting, agent loops?
3. 3How does the model perform on a fresh, never-published test of the same skill?

Applied exercise

1. 1Build a 25-question internal eval from real tasks at your company
2. 2Run the top three models against it
3. 3Compare your ranking to the public benchmark ranking
4. 4If they disagree, your eval is more relevant — trust it

The big idea: benchmark cards rank models. Your eval ranks them for your work. The second matters.