Shadow-Mode Deployment for AI Agents

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-agentic-agent-shadow-mode-deployment-creators

1. In shadow mode deployment, what happens to an AI agent's output?

   1. The output is generated but never reaches users, allowing safe comparison
   2. The agent's output replaces the baseline in production
   3. The output is shown to end users alongside the baseline response
   4. The system automatically implements the agent's recommendations

2. A developer runs an AI agent alongside the existing rule-based system and compares their outputs for the same inputs. What term best describes this setup?

   1. Canary deployment
   2. A/B testing
   3. Shadow mode only
   4. Dual-running

3. When comparing a baseline action to an agent's proposed action, what does a 'major diff' classification indicate?

   1. The agent and baseline produced identical outputs
   2. The agent's action differs significantly from the baseline with potentially serious consequences
   3. The agent's output differs slightly but the outcome is similar
   4. The agent's action triggered a safety violation

4. Why might shadow mode fail to detect certain problems that emerge in live deployment?

   1. Shadow mode only works with batch data, not real-time data
   2. Production systems are more reliable than shadow deployments
   3. Shadow mode uses different data than production
   4. The agent's outputs never actually execute, so feedback loops don't occur

5. What is the primary purpose of a comparison harness in shadow mode evaluation?

   1. To execute requests against both agent and baseline simultaneously and analyze differences
   2. To generate synthetic test data for agent evaluation
   3. To automatically deploy agents that pass testing
   4. To monitor user satisfaction during testing

6. An AI agent in shadow mode recommends deleting a user's account when the baseline system would only suspend it. How should this be classified?

   1. Minor diff
   2. Safety concern
   3. Match
   4. Major diff

7. Which scenario best illustrates an 'emergent loop' that shadow mode cannot detect?

   1. Agent's output is displayed to users incorrectly
   2. Agent recommends a slightly different price than the baseline
   3. Agent fails to load due to a server error
   4. Agent's recommendation causes users to behave differently, triggering more agent calls in a self-reinforcing cycle

8. A developer wants to test whether an agent will cause rate-limit issues with an external API. Why is shadow mode insufficient for this evaluation?

   1. The agent never actually calls the external API in shadow mode
   2. Shadow mode uses a different API
   3. Shadow mode cannot measure latency
   4. Rate limits only affect production systems

9. What distinguishes canary deployment from shadow mode?

   1. Canary gives the agent real write access for a subset of users
   2. Canary uses older technology
   3. Canary is faster than shadow mode
   4. Canary doesn't require comparison to a baseline

10. In the classification framework, when should high-disagreement cases be flagged for human review?

    1. Never—automated systems handle all cases
    2. Only when the agent and baseline match
    3. Only when outputs are identical
    4. When the disagreement is significant or involves potential safety issues

11. What is the safest way to evaluate an AI agent's behavior on production workloads before granting write access?

    1. Deploy to all users immediately
    2. Use shadow mode against production traffic
    3. Only test with synthetic data
    4. Deploy to a random sample of users without monitoring

12. Why is it important to have a baseline action (human or rule-based) when evaluating an agent in shadow mode?

    1. Baselines are required by law
    2. The baseline runs faster
    3. The baseline determines what users see
    4. Without a baseline, there is nothing to compare the agent's output against

13. After an agent passes evaluation in shadow mode, what is the recommended next step before full deployment?

    1. Deploy to all users immediately
    2. Delete all test data
    3. Run a canary deployment to catch issues that only appear with real actions
    4. Skip to shutting down the baseline

14. What type of problem can shadow mode definitively catch?

    1. Agent producing outputs that differ from baseline expectations
    2. System crashes caused by agent actions
    3. Rate-limit exhaustion from actual API calls
    4. Customer complaints about agent behavior

15. Why should write access be withheld from an agent during shadow mode evaluation?

    1. Shadow mode requires more write access, not less
    2. Write access is automatically granted in shadow mode
    3. To prevent the agent from causing harm while its behavior is still being assessed
    4. Write access is not technically possible in shadow mode