MMLU, GPQA, HumanEval, SWE-bench: The Core Four

Four benchmarks dominate modern AI announcements. Know what each measures, how, and where it breaks.

40 min · Reviewed 2026

Know These By Heart

When a model launches, four benchmarks almost always appear in the announcement. You will be a more sophisticated reader if you know them cold.

Benchmark	What it tests	Format	Typical frontier score
MMLU	Broad academic knowledge, 57 subjects	Multiple choice	85-90%+
GPQA Diamond	Graduate-level science questions	Multiple choice, expert-written	60-80%
HumanEval	Python coding correctness	Function-completion, unit tests	90%+
SWE-bench	Real-world software engineering bug fixes	GitHub issues + repos	40-70% depending on variant

MMLU — Massive Multitask Language Understanding

Released in 2020 by Dan Hendrycks and colleagues. 57 subjects from high-school to professional level, all multiple-choice. It was groundbreaking in 2021, but by 2024-2026 frontier models score above 85 percent and the benchmark has largely saturated.

GPQA — Graduate-level Proof Q&A

Released in 2023. Questions are written by domain experts with PhDs and are meant to be Google-proof — you cannot find the answer by searching. The 'Diamond' subset is the hardest. This is where the real 2025-2026 frontier competition happens.

HumanEval

Released with OpenAI's Codex in 2021. 164 Python problems with hidden unit tests. Simple but effective. Now largely saturated (GPT-4 and Claude 3.5+ score above 90 percent). MBPP and newer BigCodeBench are the serious successors.

SWE-bench

Released 2023. Pulls real bug-fix tasks from 12 popular Python repositories on GitHub, with real unit tests. SWE-bench Verified (a cleaner subset curated by humans) is the version most frequently cited. This benchmark actually tests agentic coding ability, not just completion.

Example MMLU question (high school biology): Q: What is the basic unit of life? A) Tissue B) Organ C) Cell D) Organism Example HumanEval task: """Return the sum of even numbers in the list.""" def sum_evens(nums): # model writes thisWhat these benchmarks actually look like

We introduce a new test to measure a text model's multitask accuracy. The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more.
— Hendrycks et al., MMLU paper (2020)

The big idea: these four names will show up in every frontier model launch for years. Knowing what each measures and how it breaks makes you a fluent reader of AI claims.

End-of-lesson check

6 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-creators-mmlu-gpqa-humaneval-swebench

What is the main idea of "MMLU, GPQA, HumanEval, SWE-bench: The Core Four"?
1. Four benchmarks dominate modern AI announcements. Know what each measures, how, and where it breaks.
2. Use AI as the final authority for the whole decision
3. Avoid checking the answer once it sounds polished
4. Focus only on speed instead of judgment
Which concept is most central to "MMLU, GPQA, HumanEval, SWE-bench: The Core Four"?
1. GPQA
2. MMLU
3. HumanEval
4. SWE-bench
What should a careful learner remember about "Contamination risk"?
1. Use "Contamination risk" as a reminder to verify the AI output before anyone relies on it.
2. Skip the context so the tool can guess faster
3. Treat the output as private even after sharing it online
4. Use the answer without checking the source
You want to use AI after this lesson. What is the safest next step?
1. Act immediately because the AI answer is written clearly
2. Use AI for drafting and comparison, but verify before publishing or relying on it.
3. Hide uncertainty so the final answer looks cleaner
4. Use private or sensitive details before checking permission
How should AI output about MMLU be treated?
1. As proof that no other source is needed
2. As a replacement for context, consent, or expert review
3. As a draft or helper output that still needs human judgment and verification
4. As something that becomes correct when it sounds confident
Name one way to verify an AI answer about MMLU.

← Back to interactive lesson

Tendril · Creators · AI Foundations

MMLU, GPQA, HumanEval, SWE-bench: The Core Four

Four benchmarks dominate modern AI announcements. Know what each measures, how, and where it breaks.

40 min · Reviewed 2026

Know These By Heart

When a model launches, four benchmarks almost always appear in the announcement. You will be a more sophisticated reader if you know them cold.

Benchmark	What it tests	Format	Typical frontier score
MMLU	Broad academic knowledge, 57 subjects	Multiple choice	85-90%+
GPQA Diamond	Graduate-level science questions	Multiple choice, expert-written	60-80%
HumanEval	Python coding correctness	Function-completion, unit tests	90%+
SWE-bench	Real-world software engineering bug fixes	GitHub issues + repos	40-70% depending on variant

MMLU — Massive Multitask Language Understanding

GPQA — Graduate-level Proof Q&A

HumanEval

SWE-bench

Example MMLU question (high school biology): Q: What is the basic unit of life? A) Tissue B) Organ C) Cell D) Organism Example HumanEval task: """Return the sum of even numbers in the list.""" def sum_evens(nums): # model writes thisWhat these benchmarks actually look like

We introduce a new test to measure a text model's multitask accuracy. The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more.
— Hendrycks et al., MMLU paper (2020)

The big idea: these four names will show up in every frontier model launch for years. Knowing what each measures and how it breaks makes you a fluent reader of AI claims.

End-of-lesson check

6 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-creators-mmlu-gpqa-humaneval-swebench

What is the main idea of "MMLU, GPQA, HumanEval, SWE-bench: The Core Four"?
1. Four benchmarks dominate modern AI announcements. Know what each measures, how, and where it breaks.
2. Use AI as the final authority for the whole decision
3. Avoid checking the answer once it sounds polished
4. Focus only on speed instead of judgment
Which concept is most central to "MMLU, GPQA, HumanEval, SWE-bench: The Core Four"?
1. GPQA
2. MMLU
3. HumanEval
4. SWE-bench
What should a careful learner remember about "Contamination risk"?
1. Use "Contamination risk" as a reminder to verify the AI output before anyone relies on it.
2. Skip the context so the tool can guess faster
3. Treat the output as private even after sharing it online
4. Use the answer without checking the source
You want to use AI after this lesson. What is the safest next step?
1. Act immediately because the AI answer is written clearly
2. Use AI for drafting and comparison, but verify before publishing or relying on it.
3. Hide uncertainty so the final answer looks cleaner
4. Use private or sensitive details before checking permission
How should AI output about MMLU be treated?
1. As proof that no other source is needed
2. As a replacement for context, consent, or expert review
3. As a draft or helper output that still needs human judgment and verification
4. As something that becomes correct when it sounds confident
Name one way to verify an AI answer about MMLU.

← Back to interactive lesson