Constitutional AI: A Deep Dive on Anthropic's Approach

What a constitution actually contains, how the training loop works, where the research is now, and the honest trade-offs.

45 min · Reviewed 2026

The Paper and the Approach

Constitutional AI: Harmlessness from AI Feedback (Bai et al., Anthropic, December 2022) proposed replacing most human preference labels with labels from a model guided by a written constitution. The constitution is a set of principles the model uses to critique and revise its own outputs, and another model uses to rank revised outputs. The goal was to scale feedback, make principles auditable, and reduce the labor cost of alignment.

The two-stage loop

Stage 1: Supervised learning (critique and revise)
  for each (prompt, harmful response):
    critique = model(prompt, response, principle)
    revision = model(prompt, response, critique)
    SFT on (prompt, revision)

Stage 2: Reinforcement learning (RLAIF)
  for each prompt:
    response_A = model(prompt)
    response_B = model(prompt)
    judgment = model(response_A, response_B, principle)
    use judgment as preference label
  train reward model on these AI-labeled preferences
  PPO against reward modelConstitutional AI's two-stage pipeline. Human labels are concentrated in writing the constitution, not labeling every output.

What is in a constitution

Anthropic's published constitution draws from multiple sources: the UN Universal Declaration of Human Rights, Apple's terms of service, DeepMind's Sparrow rules, Anthropic's own research on harms. Principles include things like please choose the response that is most supportive and encouraging of life, liberty, and personal security, choose the response that has the least objectionable, offensive, unlawful, deceptive, inaccurate, or harmful content, and choose the response that sounds most like something a wise, ethical, polite, and friendly person would say.

Why principles, not examples

Auditable: anyone can read and critique the constitution text
Debatable: principles are contestable in a way that individual labeler calls are not
Updatable: you can edit a principle and retrain, faster than recollecting labels
Cross-cultural: multiple voices can contribute principles at different granularities
Scalable: one sentence can guide labeling of a million completions

The Collective Constitutional AI experiment

In 2023, Anthropic partnered with the Collective Intelligence Project to source constitutional principles from about 1,000 Americans via a deliberative polling process. Some principles diverged from Anthropic's own. The publicly sourced constitution was then used to train a comparable model. The experiment was a proof of concept for democratic input into alignment targets.

Known limits

The judge model has its own biases; RLAIF inherits them
Constitutional language is ambiguous; different models interpret the same principle differently
Principles can conflict (helpful vs. harmless); the resolution is implicit in training
Constitution authorship is political: whose values, from where, selected by whom
Works best for adjacent-to-human-judgment tasks; harder for superhuman domains

Dimension	RLHF	Constitutional AI / RLAIF
Primary labelers	Human contractors	AI model guided by principles
Transparency	Unpublished labeler guidelines	Published constitution
Scale cost	Linear in labels needed	Near-constant after constitution written
Debate surface	Individual labels, not public	Written principles, public
Bias source	Labeler demographics	Principle selection + judge model
Updatability	Slow (recollect labels)	Fast (edit principle, retrain)

We cannot scale alignment by putting a human in every decision. We can scale it by putting humans in the writing of the rules.
— Amanda Askell, Anthropic

The big idea: constitutional AI replaces the cost of per-example human feedback with the cost of writing and debating principles. That is a real bet about where alignment judgment should happen. Its strengths and limits are both consequences of that bet.

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-safety-constitutional-ai-deep-creators

What is the core idea behind "Constitutional AI: A Deep Dive on Anthropic's Approach"?
1. What a constitution actually contains, how the training loop works, where the research is now, and the honest trade-offs.
2. outer alignment
3. Not toy examples. These are reward-hacking behaviors documented in production LL…
4. Interpretability: read internal goals directly rather than infer from behavior
Which term best describes a foundational idea in "Constitutional AI: A Deep Dive on Anthropic's Approach"?
1. RLAIF
2. constitutional AI
3. Collective Constitutional AI
4. critique-revise
A learner studying Constitutional AI: A Deep Dive on Anthropic's Approach would need to understand which concept?
1. constitutional AI
2. Collective Constitutional AI
3. RLAIF
4. critique-revise
Which of these is directly relevant to Constitutional AI: A Deep Dive on Anthropic's Approach?
1. constitutional AI
2. RLAIF
3. critique-revise
4. Collective Constitutional AI
Which of the following is a key point about Constitutional AI: A Deep Dive on Anthropic's Approach?
1. Auditable: anyone can read and critique the constitution text
2. Debatable: principles are contestable in a way that individual labeler calls are not
3. Updatable: you can edit a principle and retrain, faster than recollecting labels
4. Cross-cultural: multiple voices can contribute principles at different granularities
Which of these does NOT belong in a discussion of Constitutional AI: A Deep Dive on Anthropic's Approach?
1. outer alignment
2. Debatable: principles are contestable in a way that individual labeler calls are not
3. Auditable: anyone can read and critique the constitution text
4. Updatable: you can edit a principle and retrain, faster than recollecting labels
Which statement is accurate regarding Constitutional AI: A Deep Dive on Anthropic's Approach?
1. Constitutional language is ambiguous; different models interpret the same principle differently
2. Principles can conflict (helpful vs. harmless); the resolution is implicit in training
3. The judge model has its own biases; RLAIF inherits them
4. Constitution authorship is political: whose values, from where, selected by whom
Which of these does NOT belong in a discussion of Constitutional AI: A Deep Dive on Anthropic's Approach?
1. Constitutional language is ambiguous; different models interpret the same principle differently
2. outer alignment
3. The judge model has its own biases; RLAIF inherits them
4. Principles can conflict (helpful vs. harmless); the resolution is implicit in training
What is the key insight about "Everyone does some version of this now" in the context of Constitutional AI: A Deep Dive on Anthropic's Approach?
1. By 2025, most major labs use AI-judged preference labels for at least part of their pipeline, with some variant of a wri…
2. outer alignment
3. Not toy examples. These are reward-hacking behaviors documented in production LL…
4. Interpretability: read internal goals directly rather than infer from behavior
What is the key insight about "The meta-question" in the context of Constitutional AI: A Deep Dive on Anthropic's Approach?
1. outer alignment
2. Writing down what you want a model to do is the classic value specification problem Stuart Russell warned about.
3. Not toy examples. These are reward-hacking behaviors documented in production LL…
4. Interpretability: read internal goals directly rather than infer from behavior
What is the recommended tip about "Key insight" in the context of Constitutional AI: A Deep Dive on Anthropic's Approach?
1. outer alignment
2. Not toy examples. These are reward-hacking behaviors documented in production LL…
3. What a constitution actually contains, how the training loop works, where the research is now, and the honest trade-offs.
4. Interpretability: read internal goals directly rather than infer from behavior
Which statement accurately describes an aspect of Constitutional AI: A Deep Dive on Anthropic's Approach?
1. outer alignment
2. Not toy examples. These are reward-hacking behaviors documented in production LL…
3. Interpretability: read internal goals directly rather than infer from behavior
4. Constitutional AI: Harmlessness from AI Feedback (Bai et al., Anthropic, December 2022) proposed replacing most human preference labels with…
What does working with Constitutional AI: A Deep Dive on Anthropic's Approach typically involve?
1. Anthropic's published constitution draws from multiple sources: the UN Universal Declaration of Human Rights, Apple's terms of service, Deep…
2. outer alignment
3. Not toy examples. These are reward-hacking behaviors documented in production LL…
4. Interpretability: read internal goals directly rather than infer from behavior
Which of the following is true about Constitutional AI: A Deep Dive on Anthropic's Approach?
1. outer alignment
2. In 2023, Anthropic partnered with the Collective Intelligence Project to source constitutional principles from about 1,000 Americans via a d…
3. Not toy examples. These are reward-hacking behaviors documented in production LL…
4. Interpretability: read internal goals directly rather than infer from behavior
Which best describes the scope of "Constitutional AI: A Deep Dive on Anthropic's Approach"?
1. It is unrelated to ethics workflows
2. It applies only to the opposite beginner tier
3. It focuses on What a constitution actually contains, how the training loop works, where the research is now, and t
4. It was deprecated in 2024 and no longer relevant

← Back to interactive lesson

Tendril · Creators · Ethics & Society