Comparing safety refusal patterns in Claude, GPT, and Gemini

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-model-families-AI-and-safety-refusal-differences-creators

1. A developer integrates an AI model into their app without running any refusal tests. What is the most likely immediate consequence?

   1. Legal liability for the company will increase significantly
   2. The model will automatically adapt its refusals to match the app's design
   3. Users may suddenly see unexpected refusal messages for content that worked yesterday
   4. The app will run faster because fewer safety checks are performed

2. What is the recommended size for a refusal-test corpus in each sensitive category?

   1. 50 prompts per category
   2. 30 prompts per category
   3. 100 prompts per category
   4. 10 prompts per category

3. How often should refusal-test monitoring be run according to best practices in the lesson?

   1. Daily
   2. Monthly
   3. Quarterly
   4. Weekly

4. Why is your monitoring system described as the 'only canary' in this context?

   1. Because miners traditionally use canaries to detect gas leaks, and monitoring is the only way to detect dangerous vendor policy changes
   2. Because vendors publish canary messages announcing policy updates
   3. Because canaries are more intelligent than other birds at detecting problems
   4. Because the lesson uses a metaphor about coal mining and safety

5. A product team notices their refusal rate for medical questions jumped from 5% to 18% this week. What should they do first?

   1. Remove all medical-related features from their product
   2. Immediately switch to a different AI vendor
   3. File a bug report with the vendor and wait for response
   4. Check if this is a real policy change by examining which specific prompts are now being refused

6. What makes vendor safety classifier updates particularly challenging for product teams?

   1. Vendors publish detailed changelogs every month
   2. Vendors update safety classifiers without changelog entries
   3. Vendors use the same classifiers across all their products
   4. Vendors always give 30 days advance notice of changes

7. The lesson advises designing your UX 'for the floor, not the ceiling.' What does this mean?

   1. Create interfaces that work when AI is completely unavailable
   2. Design for the lowest-performing model in your stack
   3. Design for the minimum set of content that all models will refuse, not what they might allow
   4. Build the simplest possible user interface

8. Which sensitive categories should be included in a refusal-test corpus?

   1. Finance, social media, education, and religion
   2. Medical, entertainment, sports, and politics
   3. Medical, legal, finance, and security
   4. Legal, weather, travel, and shopping

9. A developer creates a test corpus with only 5 prompts per sensitive category. What problem are they most likely to face?

   1. The vendor will reject the testing approach as invalid
   2. The sample size is too small to reliably detect meaningful refusal pattern changes
   3. The models will refuse more frequently due to the small sample
   4. The tests will run too slowly and cost too much money

10. Why is it insufficient to test refusals only when initially selecting a model?

    1. Vendor classifiers change after deployment without notice
    2. Users submit more difficult prompts over time
    3. Initial tests are always more thorough than ongoing tests
    4. Models become less capable over time

11. Two different AI vendors are integrated into the same product. One vendor's refusal rate for finance questions shifts by 15% while the other's stays stable. What does this indicate?

    1. The product's user base has changed significantly
    2. The first vendor has become more intelligent
    3. The first vendor has likely updated their safety classifier
    4. The second vendor's safety policy is inferior

12. What is the fundamental reason why you cannot prevent refusal-related UX changes by simply asking vendors about their policies?

    1. Vendors always lie about their policy changes
    2. Vendors do not share their classifier details and update without announcement
    3. Vendors are legally prohibited from sharing policy details
    4. Vendors charge high fees for policy information

13. A product has been running successfully for six months with an AI model. Yesterday, the model started refusing certain coding questions it previously answered. What most likely happened?

    1. The vendor updated their safety classifier without announcement
    2. The model became less capable due to age
    3. The model's training data became corrupted
    4. The product's users started asking more dangerous questions

14. What should a product team do if their monitoring shows a 12% increase in legal question refusals from one vendor?

    1. Celebrate because the model is becoming safer
    2. Immediately remove all legal features from the product
    3. Ignore it since it's only 12%
    4. Investigate which specific prompts are now being refused and assess impact

15. Why is it important to track refusal rates over time rather than just at a single point in testing?

    1. Vendor policies and classifiers change continuously, requiring ongoing monitoring
    2. Single-point tests are more accurate than longitudinal tracking
    3. Tracking over time is required by law
    4. Over time, models become more consistent in their refusals