Scalable Oversight: Watching Models Smarter Than You

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-safety2-scalable-oversight-builders

1. What is the core issue called the 'bandwidth problem' in AI oversight?

   1. A limitation in how much data AI systems can process at once
   2. The gap between how fast AI produces output and how slowly humans can review it
   3. The delay when AI systems communicate over networks
   4. The limit on how many users can interact with an AI at the same time

2. Why might raters struggle to evaluate AI outputs in specialized domains like advanced biology or mathematics?

   1. AI systems never make errors in technical fields
   2. Those fields do not use AI systems
   3. Most raters lack the specialized expertise needed to spot errors in those domains
   4. Raters are not allowed to evaluate scientific content

3. In the 'debate' approach to scalable oversight, what do the two models do?

   1. They compete to generate the longest possible response
   2. They take turns correcting each other's grammar mistakes
   3. They work together to combine their strengths and produce one unified answer
   4. They argue opposing sides of an answer while a human judge determines which is correct

4. What does 'iterated amplification' mean as an oversight strategy?

   1. Making AI systems run faster by using more computing power
   2. Gradually increasing the number of human raters overseeing an AI
   3. Breaking complex tasks into smaller pieces that humans can check step by step
   4. Repeating the same training data multiple times to reinforce learning

5. What is 'recursive reward modeling'?

   1. Using rewards to make AI models repeat the same answers repeatedly
   2. Training a reward model on easier tasks, then using it to evaluate harder tasks
   3. Building multiple separate AI systems that never communicate with each other
   4. Training AI only on mathematical problems and nothing else

6. How does 'process supervision' differ from standard evaluation methods?

   1. It scores the reasoning steps that lead to an answer, not just whether the final answer is correct
   2. It relies entirely on other AI systems to provide feedback
   3. It gives AI systems a pass/fail grade on their overall performance
   4. It only evaluates the length of the AI's response

7. What is a 'critique model' in scalable oversight?

   1. A model that finds flaws in another model's output
   2. A model that copies and repeats training data
   3. A model that rates the creativity of AI-generated content
   4. A model that always agrees with human supervisors

8. What risk does 'rater fatigue' create in human feedback systems?

   1. Raters become too cautious and reject good AI answers
   2. AI systems become dependent on tired raters
   3. Tired raters may miss errors and accidentally reinforce incorrect AI behavior
   4. Raters start agreeing with each other too much

9. What is the main goal of scalable oversight research?

   1. Finding ways to supervise AI systems that are smarter than the humans overseeing them
   2. Replacing all human workers with AI
   3. Making AI systems run on smaller, cheaper hardware
   4. Preventing AI from generating long outputs

10. Why might RLHF alone become insufficient for very capable future AI systems?

    1. RLHF requires no human involvement
    2. Human raters cannot keep up with the complexity and length of outputs from highly capable models
    3. Human raters are always better than AI at finding errors
    4. RLHF has been proven to work perfectly for all AI systems

11. What did the lesson say about whether current scalable oversight methods are proven to work?

    1. They only work for very simple AI systems
    2. They have been proven to work perfectly on all AI systems
    3. They have been abandoned in favor of other approaches
    4. They are current best guesses, not solved methods that work for much smarter AI

12. In scalable oversight, what keeps the 'human in the loop' truly human?

    1. AI systems vote and humans only count the votes
    2. AI systems make all decisions and humans just watch
    3. Humans and AI take turns being in charge depending on the task
    4. Humans make the final judgments while AI helps extend their effective reach

13. Why is it dangerous if a model is wrong in ways the rater cannot see?

    1. The model will automatically correct itself
    2. The error will be immediately obvious to everyone
    3. Training will reinforce the error because the rater unknowingly rewards incorrect behavior
    4. The AI will refuse to generate more output

14. What does it mean to 'bootstrap supervision up the capability curve'?

    1. Start with simple AI and never attempt more advanced systems
    2. Supervise AI only at the entry level of capability
    3. Use AI to help align smarter AI, gradually building better oversight as capabilities increase
    4. Use human supervision once and then never update it

15. The lesson compares scalable oversight to a 'research bet.' What does this mean?

    1. It is an idea that has already been fully tested
    2. It is a guaranteed method that will definitely succeed
    3. It is a gamble with no reasonable basis
    4. It is a promising direction that has not yet been proven to work at scale