AI Tool Promptfoo Config Suite: Running Side-by-Side Prompt Tests

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-tools-promptfoo-config-suite-r9a4-creators

1. What is Promptfoo primarily designed to do?

   1. Deploy AI models to production servers
   2. Generate new prompts automatically based on a topic
   3. Convert natural language queries into database SQL statements
   4. Compare the same prompt against multiple AI providers to evaluate quality differences

2. When AI assists with creating a Promptfoo configuration, which task is within AI's capability?

   1. Drafting test cases and initial assertions for comparison
   2. Replacing human review of judge-graded outputs
   3. Determining what score makes a prompt acceptable to ship
   4. Deciding which provider your team should adopt permanently

3. Which three assertion types can AI help draft for Promptfoo tests?

   1. encrypt, hash, and compress
   2. contains, format, and grading-by-judge
   3. login, logout, and session
   4. deploy, scale, and latency

4. What is an 'acceptance threshold' in the context of prompt testing?

   1. A time limit for how long a test can run
   2. A configuration setting for how many providers to test simultaneously
   3. A cutoff score or criteria that determines whether a prompt passes or fails testing
   4. A file that lists which prompts to include in a test suite

5. Why must humans, not AI, decide acceptance thresholds for prompt testing?

   1. AI cannot read configuration files
   2. Thresholds are illegal to automate
   3. Thresholds require understanding of real-world requirements and risk tolerance that AI lacks
   4. AI always produces perfect scores

6. What warning does the lesson provide about judge model graders?

   1. They cannot be used with multiple providers
   2. They eliminate the need for any human review
   3. Their grading behavior can change when the underlying model is updated
   4. They always produce identical scores regardless of context

7. In Promptfoo terminology, what does 'side-by-side' testing mean?

   1. Testing prompts one after another in sequence
   2. Running identical prompts against multiple providers simultaneously to compare outputs
   3. Placing two test files next to each other in a folder
   4. Writing prompts and tests in adjacent code blocks

8. What is the recommended practice when using a judge model in Promptfoo?

   1. Pin the judge model version and periodically review its grading behavior
   2. Run judge models only in production
   3. Avoid using judge models for any reason
   4. Use the latest available version at all times

9. What does it mean to 'scaffold' a Promptfoo configuration?

   1. AI generates an initial draft with prompts, providers, tests, and assertions that humans then refine
   2. Delete existing test cases to start fresh
   3. Deploy the configuration to a live server
   4. Manually type every line of configuration from scratch

10. What can AI do regarding assertions in a Promptfoo configuration?

    1. Decide which assertions should cause permanent production blocks
    2. Write assertions that pass every test without evaluation
    3. Eliminate the need for any assertions in testing
    4. Draft initial assertions for contains, format, and judge-based grading

11. Which statement about human inspection of judge-graded outputs is correct?

    1. Humans should defer completely to the judge model's decisions
    2. Human inspection is required for every single test case, not just samples
    3. Humans must still inspect judge-graded outputs because AI cannot replace this review
    4. Human inspection is only needed for contains assertions

12. What is the prompt owner responsible for in a Promptfoo configuration created with AI assistance?

    1. Writing all test case text manually
    2. Designing the user interface of the testing tool
    3. Choosing which AI model to purchase for the company
    4. The assertions and acceptance criteria that determine if prompts are shippable

13. What information should be included when AI scaffolds a Promptfoo configuration?

    1. Prompts, providers, test cases, assertions, judge model configuration, and output format
    2. A list of竞争对手 (competitors)
    3. Just the provider API keys
    4. Only the prompts themselves

14. What is 'regression' in the context of prompt testing?

    1. A type of assertion that checks for regression testing
    2. A new feature added to the testing tool
    3. When a prompt that previously passed now fails or produces worse outputs
    4. When the testing tool crashes

15. What happens if you rely solely on AI to set acceptance thresholds without human oversight?

    1. The tests will automatically improve the prompts
    2. You might ship prompts that don't meet actual quality requirements for your users
    3. The CI pipeline will become faster
    4. There is no downside—AI is reliable for this