Prompt Evaluation and Testing: From Vibes to Rigorous Evals, Part 1

Prompt changes need measurement; a harness makes the measurement repeatable so you ship improvements with confidence.

What AI does well here

• Build representative test sets (real traffic samples + edge cases + adversarial prompts)
• Define metrics appropriate to the task (correctness, faithfulness, format compliance, safety)
• Use LLM-as-judge for scalable evaluation, calibrated against human review
• Track per-version metrics so regressions are visible

What AI cannot do

• Substitute for human evaluation on the most important behaviors
• Catch behaviors not represented in the test set
• Replace production monitoring (test set evaluation is necessary, not sufficient)

RAG quality depends on prompt design as much as retrieval quality; the prompt determines whether retrieved context actually shows up in answers.

What AI does well here

• Use structured prompt templates that separate retrieved context from user query and instructions
• Require explicit citation in answers (e.g., '[Source: doc_id, page]')
• Add 'I don't know' as an explicit option when retrieved context doesn't answer the query
• Implement post-hoc grounding checks (does every claim trace to a retrieved chunk?)

What AI cannot do

• Substitute for high-quality retrieval (bad retrieval can't be saved by good prompting)
• Eliminate hallucination entirely (it's a risk reduction, not elimination)
• Replace evaluation against ground-truth answers

Prompts are code; treating them otherwise produces undocumented changes, regressions, and outages.

What AI does well here

• Store prompts in version control (git) alongside the code that uses them
• Require code review for prompt changes the same way you review application code
• Maintain version history with rationale for each change
• Build the rollback path so reverting a prompt is as easy as reverting code

What AI cannot do

• Substitute for evaluation harness (version control doesn't tell you which version is better)
• Replace runtime A/B testing for high-stakes changes
• Make every prompt-iteration ceremonial (some need to be fast)

Undisciplined prompt iteration creates regressions; discipline (versioning, testing, review) keeps prompts production-stable.

What AI does well here

• Version prompts in source control like code
• Run evaluation suite against every change
• Code-review prompt changes the same as code changes
• Document the rationale for each change for future debugging

What AI cannot do

• Iterate prompts in production without testing
• Skip evaluation when changes feel small
• Generalize from one fix to similar prompts without testing

Eval set curation drives prompt quality; quality > quantity.

What AI does well here

• Curate from real production traffic
• Include edge cases and adversarial inputs
• Maintain ground truth where possible
• Update as use cases evolve

What AI cannot do

• Get eval coverage by adding more cases
• Substitute eval for production monitoring
• Make eval sets perfect

Prompt changes can break production; canary testing catches regressions.

What AI does well here

• Roll out prompt changes to small canary first
• Compare canary metrics to baseline
• Roll back automatically on regression
• Roll out broader after canary success

What AI cannot do

• Catch every issue in canary
• Substitute canary for actual evaluation
• Eliminate rollout risk

Prompt-level cost monitoring surfaces optimization targets; aggregate monitoring misses opportunities.

What AI does well here

• Track cost per prompt in production
• Surface high-cost prompts for review
• Generate optimization recommendations
• Maintain quality during cost optimization

What AI cannot do

• Optimize cost without measuring quality
• Eliminate token costs entirely
• Substitute monitoring for prompt design discipline

Prompt-level quality monitoring surfaces issues; aggregate metrics miss specifics.

What AI does well here

• Track quality metrics per prompt
• Surface degraded prompts for review
• Generate improvement recommendations
• Maintain prompt owner authority

What AI cannot do

• Get quality through monitoring alone
• Substitute monitoring for actual quality work
• Eliminate the maintenance burden

Most teams 'A/B test' prompts on three examples and ship the winner. Real prompt evaluation needs the same rigor as any product experiment.

What AI does well here

• Define the metric and sample size before running the test
• Use a fixed eval set large enough to detect the effect you care about
• Track variance from sampling, not just the mean
• Sanity-check with a hand-graded subset

What AI cannot do

• Detect small effects on tiny eval sets — power matters
• Substitute LLM-as-judge for human grading on all metrics
• Skip the cost-and-latency dimension of the comparison

Prompts are code — they deserve canary rollouts and the same rollback discipline.

What AI does well here

• Route a small slice of traffic to the new prompt.
• Compare key quality and cost metrics with statistical rigor.
• Auto-rollback on guardrail breach.

What AI cannot do

• Detect slow drift over weeks within a one-day canary.
• Catch issues that only appear in long conversations.

Replaying yesterday's traffic through tomorrow's prompt is the cheapest way to catch regressions.

What AI does well here

• Sample a representative slice of historical requests.
• Run baseline and candidate prompts in parallel offline.
• Generate diff reports with severity scoring.

What AI cannot do

• Capture user satisfaction without real-user feedback.
• Account for novel topics that weren't in the historical sample.

End your prompt with a numbered checklist the model must verify against, and require it to revise if any item fails.

What AI does well here

• Make implicit quality bars explicit
• Catch obvious misses inside the model
• Reduce iteration cycles

What AI cannot do

• Replace external evals
• Stop the model from hallucinating compliance
• Catch bugs the criteria didn't enumerate