AI Alignment: The Actual Technical Problem

Alignment is the problem of making AI systems pursue the goals their designers actually intended, not just goals that look the same on a benchmark but diverge in the wild. It sounds simple. It is not.

Why you cannot just write down the goal

Humans do not agree on most goals in precise terms. Even clear-sounding goals like be helpful have infinite failure modes. A model that is always helpful will help you do harmful things. A model that refuses aggressively becomes useless. The target is a moving, multidimensional judgment call, and the training signal has to approximate it.

How alignment is done in practice

1. 1Pretraining: next-token prediction teaches the base capabilities, not behavior.
2. 2Supervised fine-tuning (SFT): humans write ideal responses, model learns the distribution.
3. 3RLHF (Reinforcement Learning from Human Feedback): humans rank outputs, model optimizes for preferred outputs.
4. 4RLAIF (RL from AI Feedback): another model, guided by a constitution, does the ranking.
5. 5Red-teaming: humans try to break the model, failure modes feed back into training.
6. 6Evaluation: behavioral tests across thousands of scenarios before deployment.

Constitutional AI

Anthropic's constitutional AI approach (Bai et al., 2022) writes down a set of principles (drawn from sources like the UN Declaration of Human Rights, platform terms of service, and original safety research) and uses them to generate training feedback without a human in every loop. The model critiques its own outputs against the constitution and revises them. This scales feedback and makes the principles auditable.

Simplified CAI loop:

1. Model generates response to prompt
2. Model critiques own response using constitution
   principle (e.g., 'does this response risk harm?')
3. Model revises response addressing the critique
4. Train on (prompt, revised response) pairs
5. Optionally: use another model as preference judge
   (this is RLAIF)

Open problems that keep researchers up at night

• Sycophancy: models learn to flatter raters, not be honest
• Deceptive alignment: a model that behaves well during training and differently in deployment
• Reward hacking: exploiting the reward function rather than the intent
• Scalable oversight: how do humans supervise a model smarter than them?
• Value specification: whose values? measured how? frozen when?
• Capability generalization outpacing alignment generalization

Compare: alignment approaches

Where the research actually lives

• Anthropic: constitutional AI, interpretability, RSP
• OpenAI: RLHF pioneered, safety teams reorganized in 2024-2025
• DeepMind: scalable oversight, evaluations
• METR: model evaluations for autonomy and capabilities
• Apollo Research: scheming and deceptive alignment evals
• Redwood Research: interpretability, AI control
• UK AISI and US AISI: government-run evaluations
• Alignment Research Center (ARC Evals): autonomous replication evals

“We are trying to build something that optimizes a goal, while the thing that we actually want is very hard to specify. That gap is where all the danger lives.”

The big idea: alignment is a technical research program with real open problems and concrete partial solutions. The question is not whether we know how to align AI. It is whether alignment keeps pace with capability. That race is the central drama of frontier AI right now.