Scaling Laws: Why Bigger Worked

In 2020, researchers at OpenAI published a paper showing that language model performance follows a predictable curve. Add more parameters, more data, and more compute, and loss goes down in a smooth, mathematical way. The industry has been chasing that curve ever since.

The three dials

• Parameters (N): the number of weights in the model
• Data (D): how many tokens the model trains on
• Compute (C): total FLOPs spent during training, roughly N × D

The Chinchilla correction

The 2022 Chinchilla paper from DeepMind showed earlier models were undertrained. The optimal ratio is roughly 20 tokens of training data per parameter. Many older models had too many parameters and too little data, wasting compute.

Diminishing but not stopping

Each scaling bump costs exponentially more money. Going from GPT-3 to GPT-4 reportedly cost over a hundred million dollars. The returns are real but costly. That is why inference efficiency, mixture of experts, and better data now matter as much as pure scale.

Limits and open questions

• Public text data is running out. Synthetic and multimodal data fill the gap
• Energy and GPU supply now bottleneck progress
• Pure scaling may be hitting a ceiling without new algorithmic ideas
• Test-time compute (thinking longer at inference) is the new frontier

“The bitter lesson is that general methods that leverage computation are ultimately the most effective.”

The big idea: AI's recent leap was largely the result of executing the scaling recipe at industrial scale. Knowing the recipe demystifies both the progress and its current bottlenecks.