A/B Testing LLM Outputs

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-builders-ab-testing-llm-outputs

1. What is the core idea behind "A/B Testing LLM Outputs"?

   1. When you change a prompt, how do you know the new version is actually better? A/B testing is the honest answer.
   2. Are any expected columns missing?
   3. prompt template
   4. experiment

2. Which term best describes a foundational idea in "A/B Testing LLM Outputs"?

   1. control
   2. A/B test
   3. treatment
   4. statistical power

3. A learner studying A/B Testing LLM Outputs would need to understand which concept?

   1. A/B test
   2. treatment
   3. control
   4. statistical power

4. Which of these is directly relevant to A/B Testing LLM Outputs?

   1. A/B test
   2. control
   3. statistical power
   4. treatment

5. Which of the following is a key point about A/B Testing LLM Outputs?

   1. Pick one metric to optimize (satisfaction rating, task completion, response length, whatever matters…
   2. Define the change (new prompt, new model, new temperature)
   3. Randomly assign each incoming request to A or B
   4. Log the metric for each

6. Which of these does NOT belong in a discussion of A/B Testing LLM Outputs?

   1. Pick one metric to optimize (satisfaction rating, task completion, response length, whatever matters…
   2. Randomly assign each incoming request to A or B
   3. Are any expected columns missing?
   4. Define the change (new prompt, new model, new temperature)

7. Which statement is accurate regarding A/B Testing LLM Outputs?

   1. Confounded changes: you changed two things — you cannot tell which worked
   2. Selection bias: assigning 'easy' requests to B
   3. Peeking: checking results early and stopping when favorable (inflates false positives)
   4. Ignoring variance across users: one power user dominates the metric

8. Which of these does NOT belong in a discussion of A/B Testing LLM Outputs?

   1. Are any expected columns missing?
   2. Selection bias: assigning 'easy' requests to B
   3. Peeking: checking results early and stopping when favorable (inflates false positives)
   4. Confounded changes: you changed two things — you cannot tell which worked

9. What is the key insight about "The sample-size question" in the context of A/B Testing LLM Outputs?

   1. A rough rule: you need hundreds of requests per variant to detect a 5 percent effect with 80 percent power.
   2. Are any expected columns missing?
   3. prompt template
   4. experiment

10. What is the key insight about "Statistical vs practical significance" in the context of A/B Testing LLM Outputs?

    1. Are any expected columns missing?
    2. A 0.3% improvement with p=0.01 is statistically significant on 100K samples but practically irrelevant.
    3. prompt template
    4. experiment

11. What is the recommended tip about "Build your mental model" in the context of A/B Testing LLM Outputs?

    1. Are any expected columns missing?
    2. prompt template
    3. AI isn't magic — it's pattern recognition at scale. The more you understand how it works, the more effectively you can u…
    4. experiment

12. Which statement accurately describes an aspect of A/B Testing LLM Outputs?

    1. Are any expected columns missing?
    2. prompt template
    3. experiment
    4. An A/B test compares two variants — A is the current version, B is the proposed change.

13. What does working with A/B Testing LLM Outputs typically involve?

    1. The big idea: your prompt is not better because you think it is. A/B tests are how you find out.
    2. Are any expected columns missing?
    3. prompt template
    4. experiment

14. Which best describes the scope of "A/B Testing LLM Outputs"?

    1. It is unrelated to foundations workflows
    2. It focuses on When you change a prompt, how do you know the new version is actually better? A/B testing is the hon
    3. It applies only to the opposite beginner tier
    4. It was deprecated in 2024 and no longer relevant

15. Which section heading best belongs in a lesson about A/B Testing LLM Outputs?

    1. Are any expected columns missing?
    2. prompt template
    3. A minimal LLM A/B test
    4. experiment