Iterative Amplification

Break a hard task into smaller subtasks. Solve each with an AI helper. Combine the answers. Repeat. That is iterative amplification, a blueprint for supervising things humans can't check alone.

34 min · Reviewed 2026

Humans Plus Helpers

Paul Christiano's 2018 framework begins with a thought experiment called HCH: a Human Consulting HCH. Imagine you can summon copies of yourself to answer small subquestions, and those copies can summon more copies. In the limit, a carefully-managed tree of humans answers questions none of them could answer alone.

From HCH to trained models

Start: a human who can solve only small tasks
Build: an AI distilled to imitate the human on those small tasks
Amplify: let the human decompose a bigger task into smaller ones, each solved by the AI
Distill again: train a new AI to imitate the amplified system
Repeat: each iteration handles slightly harder problems

Known difficulties

Distillation loss: the AI never perfectly imitates the human tree
Decomposition failure: some questions resist being split into small pieces
Coordination: subtask answers may conflict in hard-to-reconcile ways
Capability overhang: the distilled model might develop shortcuts the human tree did not

The core bet is that if I can break a hard task into tasks I can do, and I can align an AI to do each one, I have aligned an AI to do the whole thing.
— Paul Christiano, Alignment Research Center

The big idea: amplification treats alignment as a property that must survive a training loop, not a gate you pass once. That framing influences a lot of modern safety work.

End-of-lesson check

8 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-safety2-iterative-amplification-creators

What is the main idea of "Iterative Amplification"?
1. Break a hard task into smaller subtasks.
2. Use AI as the final authority for the whole decision
3. Avoid checking the answer once it sounds polished
4. Focus only on speed instead of judgment
Which concept is most central to "Iterative Amplification"?
1. HCH
2. iterated amplification
3. decomposition
4. distillation
Which use of AI fits this topic best?
1. Let the AI decide what matters without your review
2. Use the answer before checking whether it fits the situation
3. Start: a human who can solve only small tasks
4. Treat the AI output as automatically correct
What should a careful learner remember about "The safety claim"?
1. Use "The safety claim" as a reminder to verify the AI output before anyone relies on it.
2. Skip the context so the tool can guess faster
3. Treat the output as private even after sharing it online
4. Use the answer without checking the source
You want to use AI after this lesson. What is the safest next step?
1. Act immediately because the AI answer is written clearly
2. AI cannot make the human values decision for you.
3. Hide uncertainty so the final answer looks cleaner
4. Use private or sensitive details before checking permission
How should AI output about iterated amplification be treated?
1. As proof that no other source is needed
2. As a replacement for context, consent, or expert review
3. As a draft or helper output that still needs human judgment and verification
4. As something that becomes correct when it sounds confident
Name one way to verify an AI answer about iterated amplification.
Which action would help you apply "Iterative Amplification" responsibly?
1. Use the tool to avoid thinking through the tradeoff
2. Keep going even if the output conflicts with a trusted source
3. Treat the AI output as automatically correct
4. Build: an AI distilled to imitate the human on those small tasks

← Back to interactive lesson

Tendril · Creators · Ethics & Society

Iterative Amplification

Break a hard task into smaller subtasks. Solve each with an AI helper. Combine the answers. Repeat. That is iterative amplification, a blueprint for supervising things humans can't check alone.

34 min · Reviewed 2026

Humans Plus Helpers

From HCH to trained models

Start: a human who can solve only small tasks
Build: an AI distilled to imitate the human on those small tasks
Amplify: let the human decompose a bigger task into smaller ones, each solved by the AI
Distill again: train a new AI to imitate the amplified system
Repeat: each iteration handles slightly harder problems

Known difficulties

Distillation loss: the AI never perfectly imitates the human tree
Decomposition failure: some questions resist being split into small pieces
Coordination: subtask answers may conflict in hard-to-reconcile ways
Capability overhang: the distilled model might develop shortcuts the human tree did not

The core bet is that if I can break a hard task into tasks I can do, and I can align an AI to do each one, I have aligned an AI to do the whole thing.
— Paul Christiano, Alignment Research Center

The big idea: amplification treats alignment as a property that must survive a training loop, not a gate you pass once. That framing influences a lot of modern safety work.

End-of-lesson check

8 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-safety2-iterative-amplification-creators

What is the main idea of "Iterative Amplification"?
1. Break a hard task into smaller subtasks.
2. Use AI as the final authority for the whole decision
3. Avoid checking the answer once it sounds polished
4. Focus only on speed instead of judgment
Which concept is most central to "Iterative Amplification"?
1. HCH
2. iterated amplification
3. decomposition
4. distillation
Which use of AI fits this topic best?
1. Let the AI decide what matters without your review
2. Use the answer before checking whether it fits the situation
3. Start: a human who can solve only small tasks
4. Treat the AI output as automatically correct
What should a careful learner remember about "The safety claim"?
1. Use "The safety claim" as a reminder to verify the AI output before anyone relies on it.
2. Skip the context so the tool can guess faster
3. Treat the output as private even after sharing it online
4. Use the answer without checking the source
You want to use AI after this lesson. What is the safest next step?
1. Act immediately because the AI answer is written clearly
2. AI cannot make the human values decision for you.
3. Hide uncertainty so the final answer looks cleaner
4. Use private or sensitive details before checking permission
How should AI output about iterated amplification be treated?
1. As proof that no other source is needed
2. As a replacement for context, consent, or expert review
3. As a draft or helper output that still needs human judgment and verification
4. As something that becomes correct when it sounds confident
Name one way to verify an AI answer about iterated amplification.
Which action would help you apply "Iterative Amplification" responsibly?
1. Use the tool to avoid thinking through the tradeoff
2. Keep going even if the output conflicts with a trusted source
3. Treat the AI output as automatically correct
4. Build: an AI distilled to imitate the human on those small tasks

← Back to interactive lesson