Scaling Laws: Why Bigger Worked

The past decade of AI progress came from a simple, ruthless law: more compute and more data, predictable improvements. Here is the math behind it.

30 min · Reviewed 2026

The Law That Changed Everything

In 2020, researchers at OpenAI published a paper showing that language model performance follows a predictable curve. Add more parameters, more data, and more compute, and loss goes down in a smooth, mathematical way. The industry has been chasing that curve ever since.

The three dials

Parameters (N): the number of weights in the model
Data (D): how many tokens the model trains on
Compute (C): total FLOPs spent during training, roughly N × D

The Chinchilla correction

The 2022 Chinchilla paper from DeepMind showed earlier models were undertrained. The optimal ratio is roughly 20 tokens of training data per parameter. Many older models had too many parameters and too little data, wasting compute.

Model	Parameters	Training tokens
GPT-3 (2020)	175B	300B
Chinchilla (2022)	70B	1.4T
Llama 3 (2024)	70B	15T
Modern frontier (2025-2026)	variable	tens of trillions

Diminishing but not stopping

Each scaling bump costs exponentially more money. Going from GPT-3 to GPT-4 reportedly cost over a hundred million dollars. The returns are real but costly. That is why inference efficiency, mixture of experts, and better data now matter as much as pure scale.

Limits and open questions

Public text data is running out. Synthetic and multimodal data fill the gap
Energy and GPU supply now bottleneck progress
Pure scaling may be hitting a ceiling without new algorithmic ideas
Test-time compute (thinking longer at inference) is the new frontier

The bitter lesson is that general methods that leverage computation are ultimately the most effective.
— Rich Sutton

The big idea: AI's recent leap was largely the result of executing the scaling recipe at industrial scale. Knowing the recipe demystifies both the progress and its current bottlenecks.

End-of-lesson check

8 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-builders-scaling-laws

What is the main idea of "Scaling Laws: Why Bigger Worked"?
1. The past decade of AI progress came from a simple, ruthless law: more compute and more data, predictable improvements. Here is the math behind it.
2. Use AI as the final authority for the whole decision
3. Avoid checking the answer once it sounds polished
4. Focus only on speed instead of judgment
Which concept is most central to "Scaling Laws: Why Bigger Worked"?
1. compute
2. scaling laws
3. parameters
4. Chinchilla
Which use of AI fits this topic best?
1. Let the AI decide what matters without your review
2. Use the answer before checking whether it fits the situation
3. Parameters (N): the number of weights in the model
4. Use the first answer without checking it
What should a careful learner remember about "Power laws, not linear"?
1. Double your compute, and loss drops by a predictable fraction, not half. The improvement gets smaller each time, but keeps coming.
2. Skip the context so the tool can guess faster
3. Treat the output as private even after sharing it online
4. Use the answer without checking the source
You want to use AI after this lesson. What is the safest next step?
1. Act immediately because the AI answer is written clearly
2. Use the AI answer as a draft, then check it against a reliable source.
3. Hide uncertainty so the final answer looks cleaner
4. Use private or sensitive details before checking permission
How should AI output about scaling laws be treated?
1. As proof that no other source is needed
2. As a replacement for context, consent, or expert review
3. As a draft or helper output that still needs human judgment and verification
4. As something that becomes correct when it sounds confident
Name one way to verify an AI answer about scaling laws.
Which action would help you apply "Scaling Laws: Why Bigger Worked" responsibly?
1. Use the tool to avoid thinking through the tradeoff
2. Keep going even if the output conflicts with a trusted source
3. Use the first answer without checking it
4. Data (D): how many tokens the model trains on

← Back to interactive lesson

Tendril · Builders · AI Foundations

Scaling Laws: Why Bigger Worked

The past decade of AI progress came from a simple, ruthless law: more compute and more data, predictable improvements. Here is the math behind it.

30 min · Reviewed 2026

The Law That Changed Everything

The three dials

Parameters (N): the number of weights in the model
Data (D): how many tokens the model trains on
Compute (C): total FLOPs spent during training, roughly N × D

The Chinchilla correction

Model	Parameters	Training tokens
GPT-3 (2020)	175B	300B
Chinchilla (2022)	70B	1.4T
Llama 3 (2024)	70B	15T
Modern frontier (2025-2026)	variable	tens of trillions

Diminishing but not stopping

Limits and open questions

Public text data is running out. Synthetic and multimodal data fill the gap
Energy and GPU supply now bottleneck progress
Pure scaling may be hitting a ceiling without new algorithmic ideas
Test-time compute (thinking longer at inference) is the new frontier

The bitter lesson is that general methods that leverage computation are ultimately the most effective.
— Rich Sutton

The big idea: AI's recent leap was largely the result of executing the scaling recipe at industrial scale. Knowing the recipe demystifies both the progress and its current bottlenecks.

End-of-lesson check

8 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-builders-scaling-laws

What is the main idea of "Scaling Laws: Why Bigger Worked"?
1. The past decade of AI progress came from a simple, ruthless law: more compute and more data, predictable improvements. Here is the math behind it.
2. Use AI as the final authority for the whole decision
3. Avoid checking the answer once it sounds polished
4. Focus only on speed instead of judgment
Which concept is most central to "Scaling Laws: Why Bigger Worked"?
1. compute
2. scaling laws
3. parameters
4. Chinchilla
Which use of AI fits this topic best?
1. Let the AI decide what matters without your review
2. Use the answer before checking whether it fits the situation
3. Parameters (N): the number of weights in the model
4. Use the first answer without checking it
What should a careful learner remember about "Power laws, not linear"?
1. Double your compute, and loss drops by a predictable fraction, not half. The improvement gets smaller each time, but keeps coming.
2. Skip the context so the tool can guess faster
3. Treat the output as private even after sharing it online
4. Use the answer without checking the source
You want to use AI after this lesson. What is the safest next step?
1. Act immediately because the AI answer is written clearly
2. Use the AI answer as a draft, then check it against a reliable source.
3. Hide uncertainty so the final answer looks cleaner
4. Use private or sensitive details before checking permission
How should AI output about scaling laws be treated?
1. As proof that no other source is needed
2. As a replacement for context, consent, or expert review
3. As a draft or helper output that still needs human judgment and verification
4. As something that becomes correct when it sounds confident
Name one way to verify an AI answer about scaling laws.
Which action would help you apply "Scaling Laws: Why Bigger Worked" responsibly?
1. Use the tool to avoid thinking through the tradeoff
2. Keep going even if the output conflicts with a trusted source
3. Use the first answer without checking it
4. Data (D): how many tokens the model trains on

← Back to interactive lesson