A/B Testing LLM Outputs

An A/B test compares two variants — A is the current version, B is the proposed change. You route half your real traffic to each, measure a metric, and see which wins. Same logic for LLM prompts, system messages, or models.

A minimal LLM A/B test

1. 1Pick one metric to optimize (satisfaction rating, task completion, response length, whatever matters)
2. 2Define the change (new prompt, new model, new temperature)
3. 3Randomly assign each incoming request to A or B
4. 4Log the metric for each
5. 5Wait for enough samples (see sample size below)
6. 6Compare means with a statistical test

Common mistakes

• Peeking: checking results early and stopping when favorable (inflates false positives)
• Confounded changes: you changed two things — you cannot tell which worked
• Selection bias: assigning 'easy' requests to B
• Ignoring variance across users: one power user dominates the metric

“Absence of evidence is not evidence of absence — especially in small A/B tests.”

The big idea: your prompt is not better because you think it is. A/B tests are how you find out.