The Three Ingredients: Data, Compute, Algorithms (Capstone)

The Ingredient Model

At the broadest possible level, every AI system in history is a mix of three ingredients. Data provides the examples. Compute runs the calculations. Algorithms decide how the calculations relate to each other. Shifting any one of them creates a new era.

The ingredients mapped to history

How the ingredients interact

• Data bottlenecks: public text ran out, driving synthetic and multimodal data
• Compute bottlenecks: GPU supply and energy availability cap scaling
• Algorithm bottlenecks: pure transformers plateau, pushing MoE, state space, and hybrid architectures
• Adding one ingredient often just exposes a shortage of another

Walkthrough: train your own tiny LLM mentally

1. Choose your data: 1B tokens of curated web + code
2. Choose your compute: 8 H100 GPUs for 72 hours
3. Choose your architecture: 1B-parameter decoder transformer with rotary embeddings and grouped-query attention
4. Tokenize the corpus with a 32k-token BPE vocabulary
5. Train with AdamW, cosine learning-rate schedule, 2048-token sequences
6. Evaluate on held-out perplexity and a small MMLU slice
7. Fine-tune on 50k human-rated instruction pairs
8. Apply RLHF or DPO for a final style pass
9. Deploy via vLLM with structured output support
10. Monitor for drift and retrain quarterly

# Minimal training config sketch config = { "model": { "layers": 24, "d_model": 2048, "n_heads": 16, "kv_heads": 4, "vocab_size": 32000, "max_seq_len": 2048, "position_embedding": "rope", }, "train": { "tokens": 20_000_000_000, # ~20 tokens per param, Chinchilla-ish "batch_size": 512, "optimizer": "adamw", "lr": 3e-4, "schedule": "cosine", }, "data": { "sources": ["fineweb", "the_stack_v2", "textbooks_synthetic"], "dedup": True, "language_filter": "en", }, }Every decision here is a bet on one of the three ingredients.

Capstone exercise

• Pick a real AI product you use (chat, code assistant, image generator).
• Estimate which ingredient is its main competitive advantage and why.
• Predict what will change first for that product: the data, the compute, or the algorithm.
• Identify what signal in the wild would confirm your prediction within 12 months.
• Write it down and revisit it later. You will be grading yourself on real-world feedback.

Everything in AI is ultimately a budget problem with three line items.

— A systems researcher

The big idea: AI is not magic. It is three ingredients, thousands of engineers, and careful attention to which ingredient is scarce this year. You now have the vocabulary, the history, and the mental model to reason about whatever comes next.