Using feature flag platforms (LaunchDarkly, Statsig) for AI rollouts

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-tools-AI-feature-flag-platforms-creators

1. When implementing feature flags for AI prompts and models, what is the recommended one-to-one mapping structure?

   1. One flag per geographic region
   2. One flag per user segment
   3. One flag per evaluation metric
   4. One flag per (prompt_version × model) combination

2. What should be the default state for a newly created feature flag controlling an AI prompt or model variant?

   1. Automatic, with gradual ramp-up
   2. Off, to prevent unintended exposure
   3. Conditional, based on user consent
   4. On, so users can immediately benefit

3. Which three metrics should be tracked for each feature flag in an AI rollout?

   1. Precision, throughput, and error rate
   2. Accuracy, F1 score, and recall
   3. Completion rate, cost, and latency
   4. User satisfaction, engagement, and retention

4. A feature flag has been running for a test period. The completion rate is 2% higher than baseline, cost is 5% lower, but latency is 15% higher. Should this flag be promoted?

   1. Yes, because two of three metrics improved
   2. No, because cost savings outweigh latency concerns
   3. No, because latency must beat baseline to promote
   4. Yes, because completion rate improved

5. What aspect of AI deployments can feature flags NOT replace, even with AI-powered analysis?

   1. Tracking completion rate and cost
   2. Traffic-level routing for canary deployments
   3. Detecting semantic drift between variants
   4. Identifying which users see which variants

6. What does 'semantic drift' refer to in the context of AI model variants behind feature flags?

   1. Cost increases due to inefficient token usage
   2. Decreasing completion rates as users lose interest
   3. Increasing latency over time as models become more complex
   4. Gradual changes in how different model versions interpret the same prompt

7. Why is scheduling regular retirements for feature flags important for long-term AI systems?

   1. To ensure newer models have priority in routing
   2. To prevent the flag configuration from becoming unmanageable
   3. To reduce monthly subscription costs for flag platforms
   4. To comply with data privacy regulations

8. Tying feature flag exposures to evaluation metrics allows teams to do what?

   1. Automatically deploy the best-performing variant to all users
   2. Reduce the cost of running multiple model variants
   3. Eliminate the need for human oversight of deployments
   4. Correlate specific prompt and model combinations with measurable outcomes

9. Why is traffic-level routing considered separate from feature flag management?

   1. Feature flags work only with LaunchDarkly, not routing infrastructure
   2. Feature flags control which code runs, not how traffic is split across versions
   3. Routing requires hardware changes while flags are software-only
   4. Routing is fully automated by modern AI systems

10. What makes feature-flagged AI deployments 'reversible'?

    1. Flags are stored in geographically distributed backup systems
    2. AI models can self-correct without human intervention
    3. The AI automatically reverts to previous versions on errors
    4. Teams can disable flags to instantly restore previous behavior

11. When running A/B tests with feature flags for different AI prompts, what is the primary operational advantage?

    1. You can isolate the effect of specific prompt or model changes
    2. You can test unlimited variants simultaneously without performance impact
    3. You reduce the need for any evaluation data
    4. You automatically get statistically significant results

12. Why should latency be tracked alongside completion rate and cost for feature flags?

    1. Latency directly correlates with model accuracy
    2. It ensures promoted features don't degrade user-perceived performance
    3. Lower latency always indicates better model quality
    4. Latency is the only metric that matters for user experience

13. What problem arises when a single feature flag controls multiple prompt versions and model variants at once?

    1. You cannot determine which combination caused observed changes
    2. Users will receive inconsistent responses
    3. The flag platform will charge extra for multiple variants
    4. The flag will automatically default to the safest option

14. What term describes the situation where accumulated, unused feature flags create a tangled configuration that is difficult to understand or modify?

    1. Technical overload
    2. Flag entropy
    3. Spaghetti configuration
    4. Flag debt

15. Compared to traditional full redeployment, how do feature flags make AI rollouts safer?

    1. They require more testing before any users see changes
    2. They automatically detect which models are failing
    3. They allow instant reversion without needing to redeploy code
    4. They prevent any code changes from reaching production