The Three Ingredients: Data, Compute, Algorithms (Capstone)

At the broadest possible level, every AI system in history is a mix of three ingredients. Data provides the examples. Compute runs the calculations. Algorithms decide how the calculations relate to each other. Shifting any one of them creates a new era.

The ingredients mapped to history

How the ingredients interact

• Data bottlenecks: public text ran out, driving synthetic and multimodal data
• Compute bottlenecks: GPU supply and energy availability cap scaling
• Algorithm bottlenecks: pure transformers plateau, pushing MoE, state space, and hybrid architectures
• Adding one ingredient often just exposes a shortage of another

Walkthrough: train your own tiny LLM mentally

1. 1Choose your data: 1B tokens of curated web + code
2. 2Choose your compute: 8 H100 GPUs for 72 hours
3. 3Choose your architecture: 1B-parameter decoder transformer with rotary embeddings and grouped-query attention
4. 4Tokenize the corpus with a 32k-token BPE vocabulary
5. 5Train with AdamW, cosine learning-rate schedule, 2048-token sequences
6. 6Evaluate on held-out perplexity and a small MMLU slice
7. 7Fine-tune on 50k human-rated instruction pairs
8. 8Apply RLHF or DPO for a final style pass
9. 9Deploy via vLLM with structured output support
10. 10Monitor for drift and retrain quarterly

# Minimal training config sketch
config = {
    "model": {
        "layers": 24,
        "d_model": 2048,
        "n_heads": 16,
        "kv_heads": 4,
        "vocab_size": 32000,
        "max_seq_len": 2048,
        "position_embedding": "rope",
    },
    "train": {
        "tokens": 20_000_000_000,  # ~20 tokens per param, Chinchilla-ish
        "batch_size": 512,
        "optimizer": "adamw",
        "lr": 3e-4,
        "schedule": "cosine",
    },
    "data": {
        "sources": ["fineweb", "the_stack_v2", "textbooks_synthetic"],
        "dedup": True,
        "language_filter": "en",
    },
}

Capstone exercise

• Pick a real AI product you use (chat, code assistant, image generator).
• Estimate which ingredient is its main competitive advantage and why.
• Predict what will change first for that product: the data, the compute, or the algorithm.
• Identify what signal in the wild would confirm your prediction within 12 months.
• Write it down and revisit it later. You will be grading yourself on real-world feedback.

“Everything in AI is ultimately a budget problem with three line items.”

The big idea: AI is not magic. It is three ingredients, thousands of engineers, and careful attention to which ingredient is scarce this year. You now have the vocabulary, the history, and the mental model to reason about whatever comes next.