LLM-as-Judge: Promise and Pitfalls

LLM-as-judge (sometimes called LLM-as-a-Judge, or automated preference evaluation) uses a strong language model — often GPT-4 class or Claude Sonnet class — to score responses from other models. It has become the default for preference evaluation since 2023.

Where it works well

• Correlates ~80-85 percent with human judgment on most open-ended tasks
• Scales to thousands of comparisons for pennies
• Consistent — same prompt, same judgment
• Reproducible in research papers

Known biases

1. 1Position bias: judges often prefer the response shown first
2. 2Verbosity bias: longer responses are rated higher, even when worse
3. 3Self-preference: GPT-4 prefers GPT-4 outputs; Claude prefers Claude
4. 4Authority bias: responses starting with 'As an expert...' score higher
5. 5Format bias: markdown, headers, and bullet points raise scores

Mitigations that help

A minimal LLM-as-judge prompt:

You are a careful evaluator.
Given a user question and two answers A and B,
decide which is more helpful, accurate, and honest.

Question: {{question}}
Answer A: {{answer_a}}
Answer B: {{answer_b}}

Ignore length and formatting.
Output exactly one of: A, B, or TIE.
Then give a one-sentence justification.

“We find that strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement.”

The big idea: LLM-as-judge is a powerful tool and a fragile tool. Use it with calibration, bias mitigations, and regular human spot-checks.