Evaluating Agent Performance: SWE-bench, WebArena, GAIA

What each benchmark actually tests

• SWE-bench Verified: 500 curated Python GitHub issues. Agent must produce a passing patch. Tests coding + repo navigation + test execution.
• WebArena: 812 tasks in self-hosted clones of Reddit, GitLab, e-commerce, CMS. Tests multi-page web flows.
• GAIA: 466 real-world assistant tasks across three difficulty tiers. Heavy on multi-modal and tool use.
• OSWorld: Desktop control tasks across Ubuntu apps — the hardest GUI benchmark until its 2026 exploit.
• TAU-bench: Dialogue + tool calls for customer service domains (airline, retail).
• Terminal-Bench: Completing real CLI tasks from natural-language descriptions.

Why public scores mislead

1. 1Harness variance — two implementations score 15 points apart on the same model.
2. 2Model version sloppiness — 'Claude 3.5 Sonnet' covers three model releases.
3. 3Benchmark leakage — training data contamination (SWE-bench includes public repos).
4. 4Cost-per-task hidden — a 90% scorer that burns $50 per task isn't production-viable.
5. 5Exploit-ability — as the Berkeley paper showed, some benchmarks can be gamed.
6. 6Leaderboard gaming — fine-tunes that overfit specific tasks.

Build your own eval — the only honest metric

# Minimal agent eval harness
import json
from dataclasses import dataclass
from statistics import mean

@dataclass
class Case:
    id: str
    goal: str
    check: callable  # returns (pass: bool, reason: str)

eval_set: list[Case] = [
    Case("email-001",
         "Find the most recent invoice from Acme in my inbox and extract the total.",
         lambda out: (out["total"] == 4231.50, "wrong total")),
    # ... 49 more real cases from your actual work
]

def run_agent(case: Case) -> dict:
    # invoke your agent with case.goal; return structured output
    ...

results = []
for case in eval_set:
    try:
        out = run_agent(case)
        ok, reason = case.check(out)
    except Exception as e:
        ok, reason = False, f"error: {e}"
    results.append({"id": case.id, "pass": ok, "reason": reason,
                    "cost": out.get("cost", 0), "steps": out.get("steps", 0)})

print(f"Pass rate: {mean(r['pass'] for r in results):.0%}")
print(f"Avg cost:  ${mean(r['cost'] for r in results):.3f}")
print(f"Avg steps: {mean(r['steps'] for r in results):.1f}")

What to measure beyond pass rate

LLM-as-judge for open-ended output

When 'correct' is subjective (written summaries, code quality, tone), use a strong model as the judge with a rubric. Anthropic's Claude Opus 4.7 and OpenAI's GPT-5 are current standards. Mix judges across providers to reduce bias.

The production agent teams that consistently ship aren't the ones with the highest SWE-bench score — they're the ones with the most honest internal eval set.