RLHF to RLAIF: How Preference Learning Scaled

A GPT-3 base model in 2020 was impressive but nearly unusable in a consumer chat interface. You had to prompt it carefully. It would helpfully suggest crimes. The jump from that to ChatGPT in late 2022 was not bigger models. It was reinforcement learning from human feedback.

The RLHF recipe

1. 1Start with a pretrained base model.
2. 2Supervised fine-tune on high-quality example completions (SFT).
3. 3Collect preference data: show humans two model outputs, ask which is better.
4. 4Train a reward model to predict human preferences from (prompt, completion) pairs.
5. 5Use PPO (or another RL algorithm) to fine-tune the policy against the reward model.
6. 6Add a KL penalty to keep the policy close to the SFT base (prevents reward hacking).

RLAIF: replace the human

RL from AI Feedback, introduced by Anthropic in Constitutional AI (Bai et al., 2022), replaces most human preference labels with labels from another (or the same) model guided by a written constitution. The model critiques outputs against principles, revises them, and does the preference ranking. You still use humans, but orders of magnitude fewer.

DPO: skip the reward model

Direct Preference Optimization (Rafailov et al., 2023) showed that you can turn preference pairs directly into a loss function without training a separate reward model and running PPO. DPO is simpler, cheaper, and often works as well or better. Many 2024-2026 open-weight alignment pipelines use DPO or variants like IPO, KTO, SimPO.

Known pathologies

• Sycophancy: Anthropic found models preference-trained on human feedback become agreeable in ways humans rate positively but that damage honesty (Sharma et al., 2023)
• Mode collapse: models produce less diverse outputs after heavy RLHF
• Over-refusal: safety training from refusal preferences generalizes to refusing benign requests
• Reward model exploitation: the policy finds inputs where the reward model gives high score but a human would not
• Distributional narrowing: RLHF shrinks the model's generative range, sometimes too much

Where the field is moving

• Process supervision (reward reasoning, not just answers) for math and code
• Rubric-based AI judges trained to apply fine-grained criteria
• Constitutional AI variants baked into every major lab's pipeline
• Online preference learning: continuously update from live user feedback
• Scalable oversight research for when the judge is weaker than the policy

“RLHF turns a pretrained model from a savant into a colleague. It also teaches the savant to please you. Separating those two is the next decade of work.”

The big idea: every modern chat model has gone through preference learning, and every one of its quirks (hedging, flattery, format bloat) traces back to how those preferences were collected. Follow the feedback and you understand the model.