AI and runbook extraction from incident transcripts: turning chaos into reusable steps

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-operations-AI-and-runbook-extraction-from-incidents-adults

1. When using AI to extract a runbook from an incident transcript, which of the following elements can AI reliably identify?

   1. The engineer's emotional state during the incident
   2. The exact sequence of commands executed during the incident
   3. The long-term financial impact of the downtime
   4. Whether the incident was caused by a hardware failure

2. Why is it important to assign an owner to an AI-extracted runbook before publishing?

   1. Without an owner, the runbook cannot be accessed by the team
   2. Ownership ensures the runbook is reviewed and updated as conditions change
   3. The AI requires an owner to function properly
   4. The owner will automatically fix any errors the AI made

3. Which of the following is a limitation of AI when extracting runbooks from incident transcripts?

   1. AI cannot determine which steps relied on an engineer's intuition
   2. AI cannot read transcript text
   3. AI cannot distinguish between technical and non-technical language
   4. AI cannot identify which commands were actually executed

4. What does the lesson identify as the primary value of extracting runbooks from incident transcripts?

   1. Replacing the need for on-call engineers
   2. Automatically fixing future incidents
   3. Capturing tribal knowledge before it fades
   4. Eliminating the need for postmortems

5. When AI flags a step as an 'improvised guess,' what should the runbook author do?

   1. Mark it clearly and add context about why it worked or might not work next time
   2. Delete the step entirely from the runbook
   3. Replace it with a random command to make the runbook more comprehensive
   4. Ignore the flag since AI flags are always inaccurate

6. Why does an extracted runbook need a review cadence?

   1. Without a cadence, the runbook will cause more incidents
   2. Runbooks become stale as systems and procedures evolve
   3. The review cadence is required by compliance regulations
   4. AI requires periodic retraining to maintain accuracy

7. Which component should be included in an AI-extracted runbook?

   1. Email addresses of all stakeholders notified
   2. A list of engineers who were on-call during the incident
   3. The hourly weather forecast for the incident duration
   4. The trigger conditions that started the incident

8. The lesson notes that AI cannot replace a postmortem facilitator. Why?

   1. Postmortems are conducted in person and AI cannot be physically present
   2. Facilitators are union members and AI cannot perform union work
   3. Postmortems require legal signatures that AI cannot provide
   4. AI cannot guide collaborative discussion or help the team learn from the incident

9. What is a 'decision point' in the context of incident runbook extraction?

   1. A database checkpoint that occurred during the incident
   2. A moment when the incident monitoring system decides to alert
   3. The exact time an engineer decided to go on break
   4. A point where the team chose between multiple response options

10. What information should be captured about dependencies in an extracted runbook?

    1. The emotional dependencies between team members
    2. Which engineers are dependent on each other
    3. External services or systems that must be available for each step to work
    4. The budget allocated to fixing the incident

11. Why might an AI-extracted runbook fail in a future incident even if it worked during the original incident?

    1. Engineers will intentionally ignore runbooks
    2. The runbook was written in the wrong programming language
    3. Conditions may be slightly different and the AI cannot validate applicability
    4. AI always makes errors in extraction

12. When is extracting a runbook from an incident transcript most valuable?

    1. Immediately after the incident while memory is fresh
    2. When the on-call engineer requests it
    3. Six months after the incident when emotions have settled
    4. Only for incidents that caused customer data loss

13. What does 'verbatim' command sequence mean in the context of runbook extraction?

    1. Commands should be sorted alphabetically
    2. Commands should be translated to pseudocode
    3. Commands should be copied exactly as they were typed
    4. Commands should be summarized in plain English

14. Why should an extracted runbook include options considered at decision points?

    1. To make the runbook longer and more comprehensive
    2. Because the AI was required to list all options
    3. To satisfy compliance requirements for documentation
    4. So future responders understand what alternatives were available and why a path was chosen

15. What is wrong with publishing an AI-extracted runbook without any review?

    1. Without review, the runbook is not legally binding
    2. Review is required by the incident management software
    3. The runbook may contain errors, outdated information, or steps that only worked due to specific circumstances
    4. The AI will charge a fee for unpublished runbooks