AI and flaky test triage

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-creators-ai-coding-AI-and-flaky-test-triage-r9a1-creators

1. Which of the following is NOT one of the four common root causes of flaky tests mentioned in this topic?

   1. Network latency and timeouts
   2. Timing and race conditions
   3. Shared mutable state between tests
   4. Database schema migrations

2. When an AI analyzes multiple failure logs from the same test, what specific task can it perform effectively?

   1. Determining which tests should be removed from the suite
   2. Grouping similar failure messages into thematic categories
   3. Running the test repeatedly to confirm if it's fixed
   4. Deciding whether to skip the test on continuous integration

3. A test depends on global mutable state that isn't reset between test runs. Which AI suggestion would be most appropriate?

   1. Skip this test on CI until someone fixes it
   2. Run the test on a different operating system
   3. Add a retry mechanism to the test runner
   4. Add a fixture reset to clean up state before each test

4. What is a hypothesis in the context of flaky test triage?

   1. An educated guess about the root cause that can be verified
   2. A statistical prediction of test pass rates
   3. A proven fact about why a test fails
   4. A description of what the test is supposed to validate

5. What limitation does AI have when it comes to confirming a fix for a flaky test?

   1. AI cannot run the test many times to verify stability
   2. AI cannot read test code
   3. AI cannot understand failure messages
   4. AI cannot identify timing issues in tests

6. What does the lesson say about AI's ability to reproduce a heisenbug?

   1. AI can easily reproduce any heisenbug by analyzing code
   2. AI can reproduce a heisenbug if given enough memory
   3. AI cannot reproduce a heisenbug it cannot run
   4. AI should try to reproduce heisenbugs by adding delays

7. When AI suggests where to add a 'wait' to fix a flaky test, what underlying issue is it likely addressing?

   1. A test ordering problem
   2. A network timeout problem
   3. A shared state problem
   4. A timing or race condition issue

8. Which scenario best describes a race condition in test flakiness?

   1. A test waits 30 seconds for a slow response
   2. A test accesses a network resource that doesn't exist
   3. Two tests modify the same global variable in unpredictable order
   4. A test fails because the database server is down

9. What does the lesson identify as something AI cannot know about tests?

   1. What root causes are most likely
   2. What failure messages mean
   3. Which tests depend on global mutable state
   4. Which tests are safe to mark as quarantined

10. A test fails intermittently when run as part of the full suite but passes when run alone. What is this an example of?

    1. A network issue
    2. A heisenbug
    3. A timing issue
    4. A test ordering problem

11. What is a minimal verification step when testing a hypothesis about flaky test causes?

    1. Run the test once to see if it passes
    2. Remove all assertions from the test
    3. Add targeted logging or a specific wait at the suspected failure point
    4. Rewrite the entire test from scratch

12. Why is 'network' listed as one of the common root causes for flaky tests?

    1. Network requests can have variable latency or fail intermittently
    2. Network tests require special hardware
    3. Network cables can become unplugged
    4. Network is never a cause of flakiness

13. What should you do after AI proposes hypotheses for why a test is flaky?

    1. Run the test repeatedly to confirm which hypothesis is correct
    2. Delete the test to avoid future issues
    3. Ask AI to fix the production code directly
    4. Immediately implement all suggested fixes

14. When AI spots a test that depends on global mutable state, what specific problem is it identifying?

    1. The test may fail if another test modified that state first
    2. The test is too slow
    3. The test requires administrator privileges
    4. The test has no assertions

15. What is the benefit of having AI rank the top 3 likely causes of test flakiness?

    1. It eliminates the need to look at failure logs
    2. It helps developers prioritize investigation efforts efficiently
    3. It automatically writes the fix for the developer
    4. It guarantees the test will pass after fixes