RLHF to RLAIF: How Preference Learning Scaled

RLHF made ChatGPT possible. RLAIF is trying to take humans out of the loop. Here is the history, the trade-offs, and where the field is going.

45 min · Reviewed 2026

The Technique That Made ChatGPT Usable

A GPT-3 base model in 2020 was impressive but nearly unusable in a consumer chat interface. You had to prompt it carefully. It would helpfully suggest crimes. The jump from that to ChatGPT in late 2022 was not bigger models. It was reinforcement learning from human feedback.

The RLHF recipe

Start with a pretrained base model.
Supervised fine-tune on high-quality example completions (SFT).
Collect preference data: show humans two model outputs, ask which is better.
Train a reward model to predict human preferences from (prompt, completion) pairs.
Use PPO (or another RL algorithm) to fine-tune the policy against the reward model.
Add a KL penalty to keep the policy close to the SFT base (prevents reward hacking).

RLAIF: replace the human

RL from AI Feedback, introduced by Anthropic in Constitutional AI (Bai et al., 2022), replaces most human preference labels with labels from another (or the same) model guided by a written constitution. The model critiques outputs against principles, revises them, and does the preference ranking. You still use humans, but orders of magnitude fewer.

DPO: skip the reward model

Direct Preference Optimization (Rafailov et al., 2023) showed that you can turn preference pairs directly into a loss function without training a separate reward model and running PPO. DPO is simpler, cheaper, and often works as well or better. Many 2024-2026 open-weight alignment pipelines use DPO or variants like IPO, KTO, SimPO.

Method	Feedback source	Loss setup	Strength	Weakness
RLHF (PPO)	Human	Reward model + PPO RL	Proven at scale	Complex, expensive, tuning-heavy
RLAIF	AI + constitution	Reward model + RL or DPO	Scales feedback cheaply	Quality depends on judge model and constitution
DPO	Human or AI	Direct preference loss	Simple, stable, no RM needed	Less mature at frontier scale
KTO	Single good/bad label	Kahneman-Tversky prospect loss	Works with one-sided data	Newer, less studied

Known pathologies

Sycophancy: Anthropic found models preference-trained on human feedback become agreeable in ways humans rate positively but that damage honesty (Sharma et al., 2023)
Mode collapse: models produce less diverse outputs after heavy RLHF
Over-refusal: safety training from refusal preferences generalizes to refusing benign requests
Reward model exploitation: the policy finds inputs where the reward model gives high score but a human would not
Distributional narrowing: RLHF shrinks the model's generative range, sometimes too much

Where the field is moving

Process supervision (reward reasoning, not just answers) for math and code
Rubric-based AI judges trained to apply fine-grained criteria
Constitutional AI variants baked into every major lab's pipeline
Online preference learning: continuously update from live user feedback
Scalable oversight research for when the judge is weaker than the policy

RLHF turns a pretrained model from a savant into a colleague. It also teaches the savant to please you. Separating those two is the next decade of work.
— An alignment researcher at OpenAI

The big idea: every modern chat model has gone through preference learning, and every one of its quirks (hedging, flattery, format bloat) traces back to how those preferences were collected. Follow the feedback and you understand the model.

End-of-lesson check

8 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-safety-rlhf-rlaif-creators

What is the main idea of "RLHF to RLAIF: How Preference Learning Scaled"?
1. RLHF made ChatGPT possible. RLAIF is trying to take humans out of the loop. Here is the history, the trade-offs, and where the field is going.
2. Use AI as the final authority for the whole decision
3. Avoid checking the answer once it sounds polished
4. Focus only on speed instead of judgment
Which concept is most central to "RLHF to RLAIF: How Preference Learning Scaled"?
1. RLAIF
2. RLHF
3. DPO
4. preference learning
Which use of AI fits this topic best?
1. Let the AI decide what matters without your review
2. Use the answer before checking whether it fits the situation
3. Start with a pretrained base model.
4. Treat the AI output as automatically correct
What should a careful learner remember about "The label bottleneck"?
1. Use "The label bottleneck" as a reminder to verify the AI output before anyone relies on it.
2. Skip the context so the tool can guess faster
3. Treat the output as private even after sharing it online
4. Use the answer without checking the source
You want to use AI after this lesson. What is the safest next step?
1. Act immediately because the AI answer is written clearly
2. AI cannot make the human values decision for you.
3. Hide uncertainty so the final answer looks cleaner
4. Use private or sensitive details before checking permission
How should AI output about RLHF be treated?
1. As proof that no other source is needed
2. As a replacement for context, consent, or expert review
3. As a draft or helper output that still needs human judgment and verification
4. As something that becomes correct when it sounds confident
Name one way to verify an AI answer about RLHF.
Which action would help you apply "RLHF to RLAIF: How Preference Learning Scaled" responsibly?
1. Use the tool to avoid thinking through the tradeoff
2. Keep going even if the output conflicts with a trusted source
3. Treat the AI output as automatically correct
4. Supervised fine-tune on high-quality example completions (SFT).

← Back to interactive lesson

Tendril · Creators · Ethics & Society