AI Evals: Testing AI Outputs Like You'd Test Code

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-tools-ai-evals-and-testing-r13a2-creators

1. What is the primary benefit of running an eval suite against a fixed test set?

   1. It allows you to compare new prompts or models against a consistent baseline
   2. It lets the AI generate entirely new test cases on its own
   3. It automatically fixes bugs in the AI model
   4. It eliminates the need for any human involvement in testing

2. According to the recommended structure, what three elements should be included in each row of an eval CSV?

   1. input, actual_output, and error_message
   2. input, expected_output, and rubric_score
   3. prompt, model_name, and timestamp
   4. question, answer_key, and difficulty_rating

3. A developer runs an eval and finds their new prompt scores 82, down from 87 with the previous prompt. What does this represent in eval terminology?

   1. A judge scoring error
   2. A successful optimization
   3. An edge case failure
   4. A performance regression

4. What bias does the lesson identify in LLM-as-judge scoring?

   1. It cannot evaluate technical content
   2. It always prefers shorter responses
   3. It only works with multiple-choice questions
   4. It prefers verbose, hedge-heavy answers

5. What does the lesson recommend to validate that LLM-as-judge scores are reliable?

   1. Only use judges trained on your specific domain
   2. Require two different judges to agree on every score
   3. Sample-validate judge scores against your own ratings
   4. Run the judge on the same data multiple times

6. Which statement best describes what AI can do well within an eval framework?

   1. Identify every possible edge case automatically
   2. Replace human judgment on subjective quality dimensions
   3. Generate test cases when seeded with examples
   4. Design the scoring rubric without any human input

7. A developer changes their prompt and wants to know if it improved the AI's responses. How does an eval help answer this question?

   1. By running the new prompt against a fixed test set and comparing rubric scores to baseline
   2. By showing the old and new prompts side-by-side to users for feedback
   3. By asking the AI to rate its own responses
   4. By measuring how long the new prompt takes to generate responses

8. The lesson states that AI cannot replace human judgment for what type of dimensions?

   1. Subjective dimensions
   2. Numerical dimensions
   3. Binary dimensions
   4. Technical dimensions

9. What is the relationship between the baseline and current scores in an eval?

   1. The baseline must always be higher than current scores
   2. The baseline is the score you hope to achieve eventually
   3. The baseline serves as a reference point for comparing new scores
   4. The baseline is calculated after running the current prompt

10. What does it mean to 'surface a regression' in the context of AI evals?

    1. To detect when a prompt or model change causes performance to drop
    2. To fix a bug that was introduced in a previous version
    3. To recover a lost version of a prompt
    4. To identify that the AI has become too verbose

11. Why might an LLM-as-judge give a high score to a response that isn't actually the best answer?

    1. Because it cannot read the response being scored
    2. Because it has access to the internet during scoring
    3. Because it prefers verbose, hedge-heavy answers that sound cautious
    4. Because it randomly assigns scores to save time

12. What is the fundamental problem that eval frameworks aim to solve?

    1. Making AI models run faster
    2. Replacing all human QA testers
    3. Eliminating the need for any test cases
    4. Turning subjective impressions like 'feels better' into measurable scores

13. After running an LLM-as-judge on your eval, you notice it consistently gives scores that seem too high. What should you do?

    1. Switch to a different scoring method entirely
    2. Sample-validate the judge scores against your own ratings
    3. Reduce the number of test cases
    4. Stop using evals and rely on user feedback

14. What must be true about a test set for it to effectively surface regressions?

    1. It must change with every test iteration
    2. It must be validated by an external auditor
    3. It must be fixed and consistent across runs
    4. It must contain only edge cases

15. Why is human judgment still necessary even when using LLM-as-judge in evals?

    1. Because subjective dimensions cannot be reliably scored by AI
    2. Because human judgment is faster than AI scoring
    3. Because evals require a human to run the test cases
    4. Because AI judges always make random errors