AI Runbook Iteration From Incidents: Closing the Gap the Outage Just Exposed

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-operations-AI-and-runbook-iteration-from-incidents-r7a2-adults

1. A team updates their runbook using AI after a payment outage. The AI identifies that Step 4 in the existing runbook was skipped during the incident response. What should happen next?

   1. The on-call team should be blamed for skipping Step 4
   2. The runbook should be updated to reflect that Step 4 was skipped and explain why it was bypassed
   3. Step 4 should be removed since it was not useful during the incident
   4. The runbook should be archived and a new one created from scratch

2. Which of the following best describes 'on-call ergonomics'?

   1. Designing runbooks and processes that minimize cognitive load for someone responding at 3am with limited context
   2. Ergonomic equipment for the on-call engineer such as standing desks
   3. A system for tracking on-call hours and overtime
   4. The physical layout of the operations center

3. An AI system compares an incident response transcript against the existing runbook and finds several mismatches. What capability is the AI demonstrating?

   1. Drift detection—identifying divergence between documented procedures and actual operational behavior
   2. Automatic incident escalation
   3. Predictive analytics for future incidents
   4. Natural language generation of new runbook content

4. A team uses AI to generate an updated runbook after a major incident, making all steps short with copy-paste commands and clear decision points. They ship the runbook without scheduling a game-day drill. What risk remains?

   1. The runbook is now too simple and lacks necessary detail
   2. The AI-generated runbook contains errors that only a game day would reveal
   3. The on-call engineer may still panic and not follow the runbook because they have not practiced using it under pressure
   4. The team has fulfilled all operational readiness requirements

5. Why are 'named decision points' important in a runbook designed for 3am readability?

   1. They speed up the runbook by eliminating unnecessary steps
   2. They are required by compliance auditors
   3. They allow the AI to automatically make decisions during incidents
   4. They give the responder a clear signal of when they need to make a choice, reducing ambiguity under stress

6. What is the fundamental limitation of AI in the runbook iteration process?

   1. AI cannot guarantee that humans will read or practice the updated runbook before the next incident
   2. AI cannot understand the technical details of the incident
   3. AI cannot access the postmortem data
   4. AI cannot generate accurate copy-paste commands

7. A postmortem reveals that the on-call engineer guessed at a command during an outage because the runbook was ambiguous. Which improvement would best address this for future incidents?

   1. Include links to additional documentation for each step
   2. Write the runbook in a more conversational tone
   3. Add more background context explaining why each step exists
   4. Replace ambiguous text with explicit copy-paste commands that remove guesswork

8. What does 'muscle memory' refer to in the context of incident response?

   1. Memory of previous incidents stored in the database
   2. Knowledge of the incident management system interface
   3. Physical hand movements for typing commands quickly
   4. The ability to execute response steps automatically through practiced repetition

9. An organization has iterated their runbook three times using AI after three separate incidents. They have not conducted a game-day exercise. What does the lesson predict about their operational readiness?

   1. They have updated documentation but may still panic during the next incident because they lack practical drill experience
   2. They need to hire more on-call engineers
   3. They should delete their old runbooks since they are obsolete
   4. They are now fully operational ready since the runbook reflects recent incidents

10. Which statement about AI's role in runbook iteration is correct?

    1. AI can predict when the next incident will occur
    2. AI can compare actual incident response steps against runbook steps and propose targeted edits
    3. AI can guarantee that on-call engineers will follow the updated runbook
    4. AI can automatically deploy runbook changes to production systems

11. A runbook step says: 'Check the error logs and determine if the issue is related to the database connection.' Why might an AI-assisted update change this step?

    1. The step should be removed because error logs are not useful
    2. The step should be expanded with more technical background
    3. The step requires interpretation and judgment at a time when responders are tired and stressed
    4. The step is too detailed and takes too long to execute

12. What is the primary goal of 'runbook iteration' as described in this lesson?

    1. Continuously narrowing the gap between documented procedures and actual incident response behavior
    2. Reducing the number of incidents that occur
    3. Replacing human on-call engineers with automated systems
    4. Creating the most comprehensive technical documentation possible

13. After an AI-assisted runbook update, a senior engineer says the team should schedule a drill before shipping the updated runbook. Why is this advice consistent with the lesson?

    1. Because runbook iteration without practice produces a panicked on-call who has not built muscle memory
    2. Because compliance requires quarterly drills
    3. Because the updated runbook is not yet approved by management
    4. Because the AI might have made errors that need human verification

14. What does it mean for a runbook to have '3am readability'?

    1. The runbook includes jokes to keep the reader awake
    2. The runbook is no longer than three pages
    3. The runbook is formatted to be read in dim lighting
    4. The runbook uses short steps, copy-paste commands, and explicit decision points to minimize cognitive load

15. An AI system flags that a runbook step was followed but produced an unexpected result. What should happen to the runbook?

    1. The runbook step should be deleted since it is unreliable
    2. No change is needed since the step was followed correctly
    3. The runbook should be updated to note the unexpected result and provide guidance for that scenario
    4. The on-call engineer should be retrained on that step