GPT-3 and the Scaling Laws

In May 2020, OpenAI published Language Models are Few-Shot Learners. GPT-3 was a Transformer with 175 billion parameters, a hundred times larger than GPT-2, trained on hundreds of billions of tokens drawn from Common Crawl, books, and Wikipedia.

The scaling laws paper

• Loss scales as a power law with parameters, data, and compute
• Larger models are more sample-efficient than smaller ones
• Optimal compute allocation favors bigger models over longer training, for a given budget
• DeepMind's Chinchilla paper in 2022 refined these laws and recommended more data relative to parameters

“Loss scales as a power law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude.”

The critical subtleties

• Scaling laws predict loss, not capabilities; capabilities emerge less predictably
• Data quality matters as much as quantity, perhaps more
• Scaling plateaus appear in specific tasks, even as average loss keeps dropping
• Cost of frontier training grew from millions to hundreds of millions of dollars

The big idea: GPT-3 plus scaling laws turned AI into a bet on scale. For a while, the bet paid off relentlessly. Whether it continues to pay off is one of the central questions of current AI research.