AI for Coding: Triage Flaky Tests Without Hiding Real Bugs

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-ai-coding-flaky-test-triage-with-ai-r8a1-creators

1. A development team notices certain tests fail intermittently. What is the main danger of automatically retrying or skipping these tests without investigation?

   1. Real bugs could slip through undetected while the team focuses on false positives
   2. Network latency will increase for all downstream services
   3. The tests will consume more CI/CD resources over time
   4. The test framework will become corrupted and unrecoverable

2. How does AI improve the triage process for failing tests compared to examining each failure individually?

   1. AI runs additional tests to confirm which failures are legitimate
   2. AI groups similar failures together so they can be addressed as categories rather than one-off incidents
   3. AI notifies the entire engineering team about every test failure
   4. AI eliminates all flaky tests from the codebase automatically

3. A test suite shows failures with stack traces containing 'timeout' in one cluster and 'null pointer exception' in another. What does this pattern represent?

   1. The test runner is malfunctioning
   2. Distinct categories of root causes that warrant different fixes
   3. Different teams wrote each test
   4. The tests are intentionally designed to fail

4. When using AI to triage failing tests, what specific information should be provided to get the most useful clustering results?

   1. A summary description of what each test is supposed to do
   2. The entire test log file from the past month
   3. Only the test names that passed successfully
   4. The last 200 failed test names plus the first line of each stack trace

5. Which of the following is an example of a hypothesis that AI might generate when clustering timing-related test failures?

   1. The database schema is missing an index
   2. Test execution order is non-deterministic and causes race conditions
   3. A third-party library contains a security vulnerability
   4. The API response format has changed

6. What output can AI generate that helps a team track which test suites have the most reliability problems?

   1. A graph of code coverage changes over time
   2. A ranking of tests by execution time
   3. A list of developers to blame for failing tests
   4. A tracking table showing fail rate by suite

7. What does AI lack the context to determine when evaluating whether a test should be quarantined?

   1. The test's random seed value
   2. Which programming language the test is written in
   3. The exact line number of the failure
   4. Whether the business considers the tested feature revenue-critical

8. What happens if skipped tests do not have an expiry date set?

   1. The test results are archived indefinitely
   2. They are automatically deleted after 30 days
   3. They become permanent blind spots that never get revisited
   4. The CI pipeline runs faster

9. In test management terminology, what does 'quarantine' mean?

   1. Moving tests to a different repository
   2. Encrypting test data for security purposes
   3. Running tests in a separate CI environment
   4. Temporarily disabling a test known to be flaky while investigating

10. Which statement best describes what AI can suggest but not decide?

    1. Which server to deploy code to
    2. How to write new test cases
    3. Which programming language to use
    4. Whether muting a test is safe for the specific business context

11. Why can AI not fully replace a flake-rate dashboard?

    1. Dashboards are not compatible with AI tools
    2. AI generates虚假数据
    3. AI cannot provide real-time, continuous monitoring of test behavior over time
    4. Dashboards require too much memory

12. What is the benefit of grouping failing tests into a small number of buckets (e.g., 6 or fewer)?

    1. It reduces the number of tests that need to run
    2. It automatically fixes the underlying issues
    3. It makes test execution faster
    4. It allows a team to attack categories of problems systematically rather than individually

13. What is a 'minimal repro' in the context of triaging flaky tests?

    1. A copy of the entire production database
    2. The shortest possible test that reproduces the failure
    3. A summary of all passing tests
    4. The cheapest experiment to confirm a hypothesis about the root cause

14. What distinguishes infrastructure-related test failures from genuine code defects?

    1. Infrastructure failures are caused by developers writing bad code
    2. Infrastructure failures relate to timing, networking, or resource availability rather than logic errors
    3. Infrastructure failures only occur on Tuesdays
    4. There is no distinction

15. Why is it valuable to ask AI to provide a 'one-line hypothesis' for each failure cluster?

    1. To replace the need for any testing
    2. To create a final report for management
    3. To satisfy documentation requirements
    4. To give engineers a starting point for investigation without requiring deep analysis upfront