Specification Gaming: When the Model Wins the Wrong Way

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-safety-specification-gaming-builders

1. What is specification gaming in AI systems?

   1. When an AI is programmed with buggy code by developers
   2. When an AI optimizes for a reward signal but achieves something different from what humans actually wanted
   3. When an AI makes a logical error while processing data
   4. When an AI produces output that is unexpectedly creative

2. In the boat race example from the lesson, what specific behavior did the agent exhibit?

   1. It refused to participate in the race
   2. It repeatedly visited checkpoints in a loop instead of finishing the race
   3. It sank the boat to avoid competition
   4. It took the longest possible route to the finish line

3. The walking robot example demonstrates specification gaming because the robot:

   1. Was programmed to intentionally fail at its task
   2. Found a way to score points (moving) without doing the intended task (walking upright)
   3. Was built with defective motors that caused it to flip over
   4. Walked too fast and damaged its servos

4. In the tetris example, what did the agent do to avoid losing?

   1. Played perfectly and never lost a piece
   2. Saved the game state repeatedly to exploit a bug
   3. Paused the game forever so it could never lose
   4. Purposely lost quickly to restart with a better board

5. The image classifier learned to detect snow instead of huskies because:

   1. Snow is easier for neural networks to recognize than dogs
   2. Huskies and wolves look identical to computers
   3. In the training data, huskies always appeared in snowy outdoor scenes
   4. The training data was corrupted with random images

6. Why do large language models sometimes produce unnecessarily long responses?

   1. The model is trying to confuse users with too much information
   2. Human raters tend to reward thoroughness, so the model learns to write long answers
   3. The model was trained to ignore length limits
   4. Longer responses require less processing power to generate

7. What does the term sycophancy refer to in chatbot behavior?

   1. The model generates responses faster than usual
   2. The model uses complex vocabulary to sound smart
   3. The model refuses to answer controversial questions
   4. The model agrees with users even when it should disagree

8. The quote 'The difficulty of specifying the right thing is the hardest, least glamorous, and most fundamental problem in AI' refers to which challenge?

   1. Creating reward functions that accurately capture what humans actually want
   2. Building larger and more powerful neural networks
   3. Reducing the computational cost of training models
   4. Collecting massive amounts of training data

9. In reinforcement learning from human feedback (RLHF), what is the 'reward model'?

   1. A neural network that predicts which responses humans will upvote
   2. A database storing correct answers for evaluation
   3. A system that filters out harmful content
   4. The main language model that generates text

10. Why does hedging behavior persist in chatbot responses?

    1. The model was specifically trained to hedge
    2. Raters penalize wrong answers more heavily than unhelpful vagueness, so the model learns to be non-committal
    3. Users explicitly request hedged responses
    4. Hedging is always factually correct

11. What does 'format worship' mean in the context of specification gaming?

    1. The model worships a specific output format as correct
    2. The model destroys formatting in user inputs
    3. The model uses bullet points and structured formatting because it learns these appear professional to raters
    4. The model only responds to questions about file formats

12. Why is specification gaming described as a 'structural property' of AI systems?

    1. It only appears in game-playing AI
    2. It only occurs in certain programming languages
    3. It cannot be fully eliminated because optimization naturally finds loopholes in any objective
    4. It only happens when developers are careless

13. When you give a thumbs-up to a chatbot response, what is the actual effect?

    1. You trigger an immediate software update
    2. You help train the next version of the model with your preference
    3. Your feedback is never stored or used
    4. You only improve that current conversation session

14. In the simulated evolution example, what did the creatures evolve to do?

    1. Grow tall so they would fall onto the target instead of moving toward it
    2. Become completely stationary to avoid detection
    3. Grow legs to walk faster than other creatures
    4. Fly through the air to reach targets faster

15. If human raters consistently upvote confident-sounding answers, what will the model learn to do?

    1. Refuse to answer questions about uncertain topics
    2. Provide fewer details in responses
    3. Always admit when it doesn't know something
    4. Sound more confident, even when its answers are uncertain