Scaling Laws and Compute-Optimal Training

Kaplan et al. (2020) showed that LLM loss follows smooth power laws in parameters, data, and compute. Hoffmann et al. (2022, the Chinchilla paper) showed that under a compute budget, you want to balance N and D rather than scaling parameters alone. These two papers shaped over a hundred billion dollars of capex.

The Kaplan power law

Loss L as a function of compute C behaves approximately like L(C) ≈ a · C^(-α) for some constants. Separately for parameters N and data D with their own exponents. The exponent α is small (around 0.05 to 0.1), meaning you need large increases in compute to get modest loss drops.

# Rough Chinchilla-optimal scaling
def compute_optimal(flops):
    # From Hoffmann et al. (2022):
    # Optimal parameters N ≈ 0.6 * sqrt(C / 6)
    # Optimal tokens    D ≈ C / (6 * N)
    N = 0.6 * (flops / 6) ** 0.5
    D = flops / (6 * N)
    return N, D

N, D = compute_optimal(1e24)  # 1e24 FLOPs budget
print(f"Params: {N:.2e}, Tokens: {D:.2e}")
# Around 20 tokens per parameter

Why Chinchilla mattered

Many modern models deliberately over-train on data well past the Chinchilla-optimal ratio. The reason: inference cost dominates total cost once a model is deployed to millions of users. A smaller, over-trained model is cheaper to serve and still captures most of the quality.

The pivot to test-time compute

By 2024-2025, pure pretraining scaling showed diminishing returns. Spending the same compute at inference — having the model think longer, sample many answers, verify, backtrack — unlocked further gains. OpenAI's o1 and o3 series, Anthropic's extended thinking, and DeepSeek's R1 all embody this inference-compute scaling law.

Bottlenecks beyond FLOPs

• Data: public high-quality text is mostly scraped. Multimodal, synthetic, and licensed data fill gaps.
• Energy: frontier training runs approach gigawatt scales
• Talent: the real throttle on how many frontier labs can exist
• Alignment: as capability scales, so does the cost of shaping behavior

What this means for your work

1. 1If you fine-tune, start with a compute-optimal base, not the largest
2. 2Benchmarks at one size often do not predict another size
3. 3Budget test-time compute explicitly in your product design
4. 4Watch for regimes where smaller, smarter beats bigger, dumber

“The exponent is small. The money spent chasing it is not.”

The big idea: scaling laws made AI progress predictable. The current frontier is learning where the curves bend, and whether new algorithms can steepen them again.