Agentic AI: Roll Out a New Agent in Shadow Mode Before Letting It Act

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-agentic-shadow-mode-rollout-r8a1-creators

1. What is the primary purpose of running an AI agent in shadow mode?

   1. To permanently replace the human workforce
   2. To train the agent using real user data
   3. To execute actions faster than the existing system
   4. To capture what the agent would do without actually performing those actions

2. Why is shadow mode described as 'the cheapest way to learn what an agent would do in production'?

   1. Because mistakes in shadow mode have no financial cost
   2. Because it eliminates the need for human developers
   3. Because it uses free open-source software
   4. Because it requires no computing power

3. In shadow mode evaluation, what does the 'agreement rate' measure?

   1. How closely the agent's proposed actions match what a human would do
   2. How many users prefer the agent over the existing system
   3. How often developers agree to deploy the agent
   4. How frequently the agent's code contains syntax errors

4. What is a 'canary' in the context of AI agent deployment?

   1. A small test group of users who try the agent first
   2. A backup system that runs if the main agent fails
   3. A type of error the agent is designed to catch
   4. A visual indicator showing agent status

5. Which issue can shadow mode reliably detect before live deployment?

   1. Whether the agent causes cache invalidation in production
   2. How users react emotionally to the agent's decisions
   3. How downstream systems respond to real actions
   4. Whether the agent would make the same choice as a human expert

6. Why can't shadow mode detect issues that 'only appear when the agent's action is real'?

   1. Because shadow mode doesn't have access to the internet
   2. Because downstream systems only respond to actual executed actions, not proposals
   3. Because the agent's code isn't running in shadow mode
   4. Because shadow mode operates in a different programming language

7. What is the purpose of a graduated rollout after shadow mode evaluation?

   1. To expose the agent to progressively larger audiences while monitoring for problems
   2. To give competitors time to prepare
   3. To limit legal liability if something goes wrong
   4. To let developers take breaks between deployment phases

8. What does it mean to 'graduate from shadow to live'?

   1. The agent's code is published as open source
   2. The agent moves from proposing actions to actually executing them in production
   3. The agent is transferred to a different development team
   4. The agent has learned enough to operate autonomously

9. Why do teams that skip shadow mode 'pay later'?

   1. They must hire additional developers
   2. They face consequences from errors that could have been caught earlier
   3. They must pay for more server time
   4. They lose customers to competitors

10. What should be captured during shadow mode to enable proper evaluation?

    1. The agent's training dataset
    2. The salaries of developers working on the agent
    3. The company's financial projections
    4. The agent's proposed actions and the human's actual actions for comparison

11. Why is user testing still necessary even after successful shadow mode evaluation?

    1. To satisfy legal requirements
    2. To reduce server costs
    3. To train the agent on new data
    4. To measure how users actually experience and respond to the agent's decisions

12. Which term describes running an AI agent alongside a human to compare their decisions without the agent acting autonomously?

    1. Canary deployment
    2. Parallel processing
    3. Redundancy
    4. Shadow mode

13. What is the primary value of surfacing disagreements between agent and human actions during shadow mode?

    1. To identify cases requiring human review before deployment
    2. To determine which developer should be fired
    3. To decide whether to use cloud or local servers
    4. To calculate how much money the agent will save

14. What type of real-world effects can only be discovered after an agent begins executing actions, not during shadow mode?

    1. The agent's memory usage patterns
    2. The agent's code syntax errors
    3. API rate limiting triggered by actual requests
    4. The agent's decision-making logic flaws

15. Why is flipping the switch directly from shadow mode to 100% live deployment risky?

    1. It violates software licensing agreements
    2. It causes server downtime
    3. Unexpected issues from real actions could affect all users simultaneously
    4. It makes the agent run slower