How an AI Model Actually Gets 'Trained' (No Math)

Modern AI models go through 3 stages: (1) Pretraining — read trillions of words of internet text; (2) Fine-tuning — read carefully curated examples of 'good' responses; (3) RLHF (Reinforcement Learning from Human Feedback) — humans rate pairs of responses and the model learns which kind people prefer. Each stage costs more than the last per data point but uses way less data.

Some examples

• GPT-4 pretraining used roughly 13 trillion tokens — more than every book ever published, plus most of the internet.
• Fine-tuning uses maybe 100,000 hand-written 'this is how to respond well' examples.
• RLHF uses ~1 million human comparisons of 'response A is better than response B.'
• Constitutional AI (Anthropic's approach) replaces some human ratings with the model rating itself against a written 'constitution' of values.

Try it!

Read Anthropic's Constitutional AI paper summary on their website (no math, just plain English about how they trained Claude). It takes 15 minutes and you'll understand more about modern AI than 95% of people.