Agent On-Call Rotation: Who Wakes Up When Agents Fail

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-agentic-agent-on-call-rotation-creators

1. What is the primary reason standard infrastructure on-call teams cannot fully cover AI agent operations?

   1. Infrastructure on-call uses different paging systems than AI agents
   2. AI agents exhibit failure modes that require ML expertise to diagnose and resolve
   3. Traditional infra failures are more severe than AI failures
   4. Standard infra teams lack access to model weights and training data

2. A production AI agent suddenly begins returning nonsensical responses that are technically valid but contextually incorrect. Which failure mode best describes this?

   1. Cost spike anomaly
   2. Infrastructure timeout
   3. Model degradation
   4. Rate limit exhaustion

3. Why are runbooks important for AI agent on-call rotations?

   1. Runbooks are only needed for infrastructure, not AI systems
   2. Runbooks automatically fix failing agents without human intervention
   3. Runbooks eliminate the need for trained on-call engineers
   4. Runbooks provide step-by-step procedures for handling specific AI failure scenarios

4. A company operates AI agents serving users across Tokyo, London, and San Francisco. What is the primary reason they need time zone coverage in their on-call rotation?

   1. Failure incidents are more likely in certain time zones
   2. AI agents perform better during local business hours
   3. Time zone coverage is required by regulatory compliance
   4. Users expect immediate response regardless of when issues occur

5. What does cross-discipline training for AI agent on-call teams involve?

   1. Teaching operations personnel and ML engineers each other's domains
   2. Requiring all on-call staff to become machine learning researchers
   3. Replacing operations training with ML training
   4. Training operators to only monitor ML systems during incidents

6. What inherent limitation prevents AI from eliminating the cost of 24/7 on-call coverage for agent operations?

   1. AI cannot access incident management systems
   2. Regulations forbid fully automated incident response
   3. Human judgment is required for novel failures that fall outside training data
   4. AI systems require more maintenance than traditional software

7. An AI agent begins generating dramatically higher costs per hour due to excessive retry logic. Which failure mode does this represent?

   1. Rate limit exhaustion
   2. Training data corruption
   3. Model degradation
   4. Cost spike anomaly

8. After resolving an AI agent incident, what should the post-incident review process primarily focus on?

   1. Calculating the exact financial loss from the incident
   2. Updating the model's training data with failed outputs
   3. Identifying patterns to improve runbooks and prevent recurrence
   4. Assigning blame to the engineer who was on-call

9. A third-party AI API begins returning 429 errors to your agent. Which failure mode is occurring?

   1. Model degradation
   2. Rate limit exhaustion
   3. Cost spike anomaly
   4. Training data drift

10. What is the main purpose of escalation paths in an AI agent on-call rotation?

    1. To provide a clear hierarchy for when issues require additional expertise or authority
    2. To eliminate the need for on-call engineers
    3. To ensure incidents are resolved by the most senior engineer available
    4. To automatically route all incidents to management

11. When designing an on-call rotation for production AI agents, which factor should be prioritized beyond simple availability?

    1. Reducing on-call compensation costs
    2. Scheduling only senior engineers for off-hours
    3. Ensuring coverage across time zones with appropriate expertise
    4. Minimizing the number of engineers on-call

12. What distinguishes AI-specific runbooks from standard infrastructure runbooks?

    1. AI runbooks address ML-specific failure modes like model degradation and cost anomalies
    2. Infrastructure runbooks are more important than AI runbooks
    3. There is no meaningful difference between them
    4. AI runbooks are longer and more detailed

13. Which challenge is unique to AI agent on-call compared to traditional software on-call?

    1. AI behavior can degrade in ways that are difficult to detect without specialized monitoring
    2. Hardware failures do not occur in AI systems
    3. AI agents require constant human monitoring
    4. Traditional software does not require any on-call coverage

14. Why is it impossible for AI to fully predict every failure in agent operations?

    1. AI systems are not advanced enough to monitor themselves
    2. AI researchers have not yet discovered how to predict failures
    3. Prediction is technically possible but economically unfeasible
    4. Real-world inputs continuously produce novel combinations outside training data

15. Why is staffing AI agent on-call with only operations personnel insufficient?

    1. AI failures often require ML expertise to diagnose root causes
    2. Regulations require ML PhDs for all AI on-call
    3. Operations personnel are too expensive for on-call
    4. Operations teams refuse to take on-call shifts for AI systems