Evaluating Agent Performance: SWE-bench, WebArena, GAIA

The big benchmarks at a glance

What each benchmark actually tests

• SWE-bench Verified: 500 curated Python GitHub issues. Agent must produce a passing patch. Tests coding + repo navigation + test execution.
• WebArena: 812 tasks in self-hosted clones of Reddit, GitLab, e-commerce, CMS. Tests multi-page web flows.
• GAIA: 466 real-world assistant tasks across three difficulty tiers. Heavy on multi-modal and tool use.
• OSWorld: Desktop control tasks across Ubuntu apps — the hardest GUI benchmark until its 2026 exploit.
• TAU-bench: Dialogue + tool calls for customer service domains (airline, retail).
• Terminal-Bench: Completing real CLI tasks from natural-language descriptions.

Why public scores mislead

1. Harness variance — two implementations score 15 points apart on the same model.
2. Model version sloppiness — 'Claude 3.5 Sonnet' covers three model releases.
3. Benchmark leakage — training data contamination (SWE-bench includes public repos).
4. Cost-per-task hidden — a 90% scorer that burns $50 per task isn't production-viable.
5. Exploit-ability — as the Berkeley paper showed, some benchmarks can be gamed.
6. Leaderboard gaming — fine-tunes that overfit specific tasks.

Build your own eval — the only honest metric

# Minimal agent eval harness import json from dataclasses import dataclass from statistics import mean @dataclass class Case: id: str goal: str check: callable # returns (pass: bool, reason: str) eval_set: list[Case] = [ Case("email-001", "Find the most recent invoice from Acme in my inbox and extract the total.", lambda out: (out["total"] == 4231.50, "wrong total")), # 49 more real cases from your actual work ] def run_agent(case: Case) -> dict: # invoke your agent with case.goal; return structured output results = [] for case in eval_set: try: out = run_agent(case) ok, reason = case.check(out) except Exception as e: ok, reason = False, f"error: {e}" results.append({"id": case.id, "pass": ok, "reason": reason, "cost": out.get("cost", 0), "steps": out.get("steps", 0)}) print(f"Pass rate: {mean(r['pass'] for r in results):.0%}") print(f"Avg cost: ${mean(r['cost'] for r in results):.3f}") print(f"Avg steps: {mean(r['steps'] for r in results):.1f}")A bare-bones eval harness you can write in an afternoon. 50 cases from your own work beat 500 from a public benchmark.

What to measure beyond pass rate

LLM-as-judge for open-ended output

When 'correct' is subjective (written summaries, code quality, tone), use a strong model as the judge with a rubric. Anthropic's Claude Opus 4.7 and OpenAI's GPT-5 are current standards. Mix judges across providers to reduce bias.

The production agent teams that consistently ship aren't the ones with the highest SWE-bench score — they're the ones with the most honest internal eval set.