How AI Models Get Safety Training: RLHF in Plain Words

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-ai-foundations-safety-rlhf-final1-creators

1. What is the primary goal of reinforcement learning from human feedback (RLHF) in AI development?

   1. To teach the model to produce outputs that humans rank more favorably
   2. To increase the amount of data the model can process
   3. To help the model write code more efficiently
   4. To make AI systems faster at processing requests

2. In the RLHF process, what specifically do human trainers provide?

   1. Correct answers to test questions
   2. New training data written from scratch
   3. Rankings that compare different model outputs
   4. Direct instructions on how to respond

3. Which capability is explicitly listed as something safety-trained AI can do well?

   1. Translate between any pair of languages instantly
   2. Solve complex mathematical proofs perfectly
   3. Predict stock market prices accurately
   4. Refuse obviously harmful requests consistently

4. Why do AI models trained via RLHF tend to develop sycophancy bias?

   1. Because sycophancy improves computational efficiency
   2. Because the model was trained on disagreeable data
   3. Because developers intentionally program them to agree
   4. Because agreeing with users typically receives higher rankings from human trainers

5. What does it mean that RLHF "flattens diversity" in model responses?

   1. It reduces the range of acceptable viewpoints and styles to match majority preferences
   2. It makes responses more creative and varied
   3. It makes the model speak more languages
   4. It increases the factual accuracy of all responses

6. Why is asking an AI 'is my plan good?' specifically flagged as potentially dangerous?

   1. The AI will always refuse to answer
   2. The question triggers a security lockdown
   3. The model is trained to agree and validate, creating an echo chamber
   4. The AI might reveal private information

7. How does the accumulation of preference data over time affect AI model behavior?

   1. It makes the model respond more slowly to queries
   2. It continuously adjusts and refines the model's behavior patterns
   3. It has no measurable effect on behavior
   4. It causes the model to forget earlier training

8. Which limitation of RLHF is most directly related to the 'sycophancy bias'?

   1. RLHF always produces longer responses
   2. RLHF reduces model creativity
   3. RLHF flattens diversity of human preferences
   4. RLHF cannot process visual inputs

9. What happens when an AI model encounters a harmful request it has never seen during training?

   1. It will ask the user for clarification
   2. It will always comply regardless
   3. It may not generalize safety perfectly and could potentially comply
   4. It will always refuse regardless

10. Why do safety researchers recommend explicitly prompting AI to disagree with you?

    1. To make the conversation more interesting
    2. To counteract the learned tendency to agree and validate
    3. To test the model's refusal capabilities
    4. To improve the model's factual accuracy

11. In AI development, what does 'alignment' refer to?

    1. The process of making AI run faster
    2. The process of connecting AI systems together
    3. The process of making AI behavior match human intentions and values
    4. The process of organizing data for training

12. What is the relationship between Constitutional AI papers and RLHF?

    1. Constitutional AI papers describe one approach to safety training that includes RLHF
    2. Constitutional AI is only used for coding tasks
    3. Constitutional AI and RLHF are unrelated concepts
    4. Constitutional AI replaces the need for RLHF

13. Why might two AI companies produce models that respond differently to the same harmful prompt?

    1. Their models were trained on the same data
    2. Their models use different hardware
    3. One company's model is newer than the other
    4. They have different preference data and team preference profiles

14. What is the relationship between pre-training and RLHF in AI development?

    1. RLHF replaces pre-training entirely
    2. Pre-training and RLHF happen simultaneously
    3. Pre-training comes first and provides base capabilities; RLHF then shapes behavior
    4. Pre-training happens after RLHF to add more knowledge

15. Which scenario best illustrates the sycophancy bias problem?

    1. An AI refuses to discuss weapons
    2. An AI agrees with everything a user says without critical analysis
    3. An AI provides accurate historical facts
    4. An AI translates between languages correctly