Deterministic replay tests for non-deterministic AI agents

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-agentic-agent-deterministic-replay-test-creators

1. In deterministic replay testing, what is the primary purpose of recording a conversation?

   1. To capture a specific execution trace that can be replayed in CI to verify consistent behavior
   2. To demonstrate the AI's capabilities to stakeholders
   3. To train the AI model on new examples
   4. To enable tests to run without network connectivity

2. What is a fixture in the context of deterministic replay testing?

   1. A type of assertion that compares AI outputs
   2. A recorded LLM response stored and retrieved by message hash during test execution
   3. A configuration file for CI pipelines
   4. A testing framework for web applications

3. What three components make up a complete test case in deterministic replay testing?

   1. User ID, session token, API key
   2. Input transcript, expected tool sequence, expected final state
   3. Model name, temperature setting, max tokens
   4. Input prompt, expected token count, response time

4. Why are recordings alone insufficient as a complete testing strategy for AI agents?

   1. Recordings cannot be version controlled
   2. Recordings consume too much storage space
   3. Recordings make tests run slower
   4. Recordings cannot detect when the agent behaves correctly for novel inputs it has never seen

5. What happens to replay test recordings when the system prompt is modified?

   1. They become more accurate because the AI has been improved
   2. They are deleted to save storage space
   3. They are automatically updated to match the new prompt
   4. They become stale and may produce false negatives when the agent's actual behavior differs from the recording

6. What mechanism should be implemented to handle stale recordings after system prompt changes?

   1. Ignoring prompt differences to maintain test stability
   2. Automatic flagging of tests where the recorded prompt differs from current code
   3. Manual review of all test recordings
   4. Deletion of all recordings older than 30 days

7. What does diffing tool-call sequences help detect in replay testing?

   1. Differences in response latency
   2. Regressions in agent behavior that cause different tool execution order or selection
   3. Memory leaks in the testing framework
   4. Changes in the AI model's vocabulary

8. In replay testing, how are actual LLM calls replaced during test execution?

   1. With random number generators
   2. With hardcoded string literals
   3. With mocked HTTP responses
   4. With recorded responses keyed by message hash

9. What is the relationship between replay testing and continuous integration (CI)?

   1. Replay tests cannot run in CI environments
   2. Replay tests run in CI to catch behavior changes without being affected by model output jitter
   3. Replay tests only run on developer workstations
   4. Replay tests require manual execution in CI

10. A developer modifies the tool definitions available to an agent. What is the most appropriate next step for the replay test suite?

    1. Review and likely re-record tests that depend on the modified tool definitions
    2. Delete all existing recordings and start fresh
    3. Increase the temperature setting to increase test coverage
    4. No changes needed - replay tests will automatically adapt

11. What problem does deterministic replay testing specifically solve for AI agents?

    1. It reduces the cost of API calls to zero
    2. It eliminates the need for any test automation
    3. It enables reliable, automated regression testing despite inherent randomness
    4. It makes stochastic agents produce deterministic outputs

12. What is 'model jitter' in the context of AI agent testing?

    1. Random variations in API response times
    2. Memory leaks causing test instability
    3. Inherent randomness in model outputs causing different results for identical inputs
    4. Physical movement of server hardware

13. What type of failures can replay testing definitively catch?

    1. Behavioral regressions in previously recorded scenarios
    2. Failures caused by inputs the agent has never encountered before
    3. Failures that occur only under specific network conditions
    4. Failures related to UI rendering in edge browsers

14. Why is it important that fixtures are keyed by message hash rather than just stored sequentially?

    1. Hash keys allow correct fixture retrieval regardless of test execution order or message position
    2. Hash keys use less storage space
    3. Hash keys are required by CI pipelines
    4. Sequential keys are not supported by modern testing frameworks

15. A team wants to ensure their replay tests remain valid over time. What practice should they adopt?

    1. Using the latest AI model available
    2. Automatically flagging tests when recorded prompts differ from current code
    3. Avoiding any changes to the agent after recording
    4. Running tests only once per year