The Three Ingredients: Data, Compute, Algorithms (Capstone)

Every AI breakthrough of the past decade rests on three interacting ingredients. Synthesize everything you have learned into one working model.

55 min · Reviewed 2026

The Ingredient Model

At the broadest possible level, every AI system in history is a mix of three ingredients. Data provides the examples. Compute runs the calculations. Algorithms decide how the calculations relate to each other. Shifting any one of them creates a new era.

The ingredients mapped to history

Era	Data	Compute	Algorithm
1980s expert systems	Handwritten rules	Desktop CPUs	Rule-based inference
2000s statistical ML	Small curated datasets	Commodity servers	SVM, decision trees
2010s deep learning	ImageNet-scale labeled sets	GPUs in parallel	Convolutional and recurrent nets
2020s LLM era	Internet-scale text + human feedback	Datacenter-scale clusters	Transformer with RLHF
2025-2026 reasoning era	Synthetic + verifiable traces	Test-time inference scaling	RL on self-generated reasoning

How the ingredients interact

Data bottlenecks: public text ran out, driving synthetic and multimodal data
Compute bottlenecks: GPU supply and energy availability cap scaling
Algorithm bottlenecks: pure transformers plateau, pushing MoE, state space, and hybrid architectures
Adding one ingredient often just exposes a shortage of another

Walkthrough: train your own tiny LLM mentally

Choose your data: 1B tokens of curated web + code
Choose your compute: 8 H100 GPUs for 72 hours
Choose your architecture: 1B-parameter decoder transformer with rotary embeddings and grouped-query attention
Tokenize the corpus with a 32k-token BPE vocabulary
Train with AdamW, cosine learning-rate schedule, 2048-token sequences
Evaluate on held-out perplexity and a small MMLU slice
Fine-tune on 50k human-rated instruction pairs
Apply RLHF or DPO for a final style pass
Deploy via vLLM with structured output support
Monitor for drift and retrain quarterly

# Minimal training config sketch
config = {
    "model": {
        "layers": 24,
        "d_model": 2048,
        "n_heads": 16,
        "kv_heads": 4,
        "vocab_size": 32000,
        "max_seq_len": 2048,
        "position_embedding": "rope",
    },
    "train": {
        "tokens": 20_000_000_000,  # ~20 tokens per param, Chinchilla-ish
        "batch_size": 512,
        "optimizer": "adamw",
        "lr": 3e-4,
        "schedule": "cosine",
    },
    "data": {
        "sources": ["fineweb", "the_stack_v2", "textbooks_synthetic"],
        "dedup": True,
        "language_filter": "en",
    },
}Every decision here is a bet on one of the three ingredients.

Capstone exercise

Pick a real AI product you use (chat, code assistant, image generator).
Estimate which ingredient is its main competitive advantage and why.
Predict what will change first for that product: the data, the compute, or the algorithm.
Identify what signal in the wild would confirm your prediction within 12 months.
Write it down and revisit it later. You will be grading yourself on real-world feedback.

Everything in AI is ultimately a budget problem with three line items.
— A systems researcher

The big idea: AI is not magic. It is three ingredients, thousands of engineers, and careful attention to which ingredient is scarce this year. You now have the vocabulary, the history, and the mental model to reason about whatever comes next.

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-creators-three-ingredients-capstone

What is the core idea behind "The Three Ingredients: Data, Compute, Algorithms (Capstone)"?
1. Every AI breakthrough of the past decade rests on three interacting ingredients. Synthesize everything you have learned into one working model.
2. It is like teaching a smart kid your house rules.
3. Make every existing tool magically MCP-compatible — adoption takes time
4. Speed up training and inference 2-4x on modern GPUs
Which term best describes a foundational idea in "The Three Ingredients: Data, Compute, Algorithms (Capstone)"?
1. compute
2. data
3. algorithm
4. bitter lesson
A learner studying The Three Ingredients: Data, Compute, Algorithms (Capstone) would need to understand which concept?
1. data
2. algorithm
3. compute
4. bitter lesson
Which of these is directly relevant to The Three Ingredients: Data, Compute, Algorithms (Capstone)?
1. data
2. compute
3. bitter lesson
4. algorithm
Which of the following is a key point about The Three Ingredients: Data, Compute, Algorithms (Capstone)?
1. Data bottlenecks: public text ran out, driving synthetic and multimodal data
2. Compute bottlenecks: GPU supply and energy availability cap scaling
3. Algorithm bottlenecks: pure transformers plateau, pushing MoE, state space, and hybrid architectures
4. Adding one ingredient often just exposes a shortage of another
Which of these does NOT belong in a discussion of The Three Ingredients: Data, Compute, Algorithms (Capstone)?
1. Algorithm bottlenecks: pure transformers plateau, pushing MoE, state space, and hybrid architectures
2. It is like teaching a smart kid your house rules.
3. Compute bottlenecks: GPU supply and energy availability cap scaling
4. Data bottlenecks: public text ran out, driving synthetic and multimodal data
Which statement is accurate regarding The Three Ingredients: Data, Compute, Algorithms (Capstone)?
1. Choose your compute: 8 H100 GPUs for 72 hours
2. Choose your architecture: 1B-parameter decoder transformer with rotary embeddings and grouped-query …
3. Choose your data: 1B tokens of curated web + code
4. Tokenize the corpus with a 32k-token BPE vocabulary
Which of these does NOT belong in a discussion of The Three Ingredients: Data, Compute, Algorithms (Capstone)?
1. Choose your architecture: 1B-parameter decoder transformer with rotary embeddings and grouped-query …
2. Choose your compute: 8 H100 GPUs for 72 hours
3. It is like teaching a smart kid your house rules.
4. Choose your data: 1B tokens of curated web + code
What is the key insight about "Bitter lesson revisited" in the context of The Three Ingredients: Data, Compute, Algorithms (Capstone)?
1. Rich Sutton's bitter lesson says general methods plus compute beat clever methods plus human priors.
2. It is like teaching a smart kid your house rules.
3. Make every existing tool magically MCP-compatible — adoption takes time
4. Speed up training and inference 2-4x on modern GPUs
What is the recommended tip about "Ground your practice in fundamentals" in the context of The Three Ingredients: Data, Compute, Algorithms (Capstone)?
1. It is like teaching a smart kid your house rules.
2. Every AI capability has an underlying mechanism. Understanding that mechanism tells you where it'll fail — which is more…
3. Make every existing tool magically MCP-compatible — adoption takes time
4. Speed up training and inference 2-4x on modern GPUs
Which statement accurately describes an aspect of The Three Ingredients: Data, Compute, Algorithms (Capstone)?
1. It is like teaching a smart kid your house rules.
2. Make every existing tool magically MCP-compatible — adoption takes time
3. At the broadest possible level, every AI system in history is a mix of three ingredients. Data provides the examples.
4. Speed up training and inference 2-4x on modern GPUs
What does working with The Three Ingredients: Data, Compute, Algorithms (Capstone) typically involve?
1. It is like teaching a smart kid your house rules.
2. Make every existing tool magically MCP-compatible — adoption takes time
3. Speed up training and inference 2-4x on modern GPUs
4. The big idea: AI is not magic. It is three ingredients, thousands of engineers, and careful attention to which ingredient is scarce this yea…
Which best describes the scope of "The Three Ingredients: Data, Compute, Algorithms (Capstone)"?
1. It focuses on Every AI breakthrough of the past decade rests on three interacting ingredients. Synthesize everythi
2. It is unrelated to foundations workflows
3. It applies only to the opposite beginner tier
4. It was deprecated in 2024 and no longer relevant
Which section heading best belongs in a lesson about The Three Ingredients: Data, Compute, Algorithms (Capstone)?
1. It is like teaching a smart kid your house rules.
2. The ingredients mapped to history
3. Make every existing tool magically MCP-compatible — adoption takes time
4. Speed up training and inference 2-4x on modern GPUs
Which section heading best belongs in a lesson about The Three Ingredients: Data, Compute, Algorithms (Capstone)?
1. It is like teaching a smart kid your house rules.
2. Make every existing tool magically MCP-compatible — adoption takes time
3. How the ingredients interact
4. Speed up training and inference 2-4x on modern GPUs

← Back to interactive lesson

Tendril · Creators · AI Foundations

The Three Ingredients: Data, Compute, Algorithms (Capstone)

Every AI breakthrough of the past decade rests on three interacting ingredients. Synthesize everything you have learned into one working model.

55 min · Reviewed 2026

The Ingredient Model

The ingredients mapped to history

Era	Data	Compute	Algorithm
1980s expert systems	Handwritten rules	Desktop CPUs	Rule-based inference
2000s statistical ML	Small curated datasets	Commodity servers	SVM, decision trees
2010s deep learning	ImageNet-scale labeled sets	GPUs in parallel	Convolutional and recurrent nets
2020s LLM era	Internet-scale text + human feedback	Datacenter-scale clusters	Transformer with RLHF
2025-2026 reasoning era	Synthetic + verifiable traces	Test-time inference scaling	RL on self-generated reasoning

How the ingredients interact

Data bottlenecks: public text ran out, driving synthetic and multimodal data
Compute bottlenecks: GPU supply and energy availability cap scaling
Algorithm bottlenecks: pure transformers plateau, pushing MoE, state space, and hybrid architectures
Adding one ingredient often just exposes a shortage of another

Walkthrough: train your own tiny LLM mentally

Choose your data: 1B tokens of curated web + code
Choose your compute: 8 H100 GPUs for 72 hours
Choose your architecture: 1B-parameter decoder transformer with rotary embeddings and grouped-query attention
Tokenize the corpus with a 32k-token BPE vocabulary
Train with AdamW, cosine learning-rate schedule, 2048-token sequences
Evaluate on held-out perplexity and a small MMLU slice
Fine-tune on 50k human-rated instruction pairs
Apply RLHF or DPO for a final style pass
Deploy via vLLM with structured output support
Monitor for drift and retrain quarterly

# Minimal training config sketch
config = {
    "model": {
        "layers": 24,
        "d_model": 2048,
        "n_heads": 16,
        "kv_heads": 4,
        "vocab_size": 32000,
        "max_seq_len": 2048,
        "position_embedding": "rope",
    },
    "train": {
        "tokens": 20_000_000_000,  # ~20 tokens per param, Chinchilla-ish
        "batch_size": 512,
        "optimizer": "adamw",
        "lr": 3e-4,
        "schedule": "cosine",
    },
    "data": {
        "sources": ["fineweb", "the_stack_v2", "textbooks_synthetic"],
        "dedup": True,
        "language_filter": "en",
    },
}Every decision here is a bet on one of the three ingredients.

Capstone exercise

Pick a real AI product you use (chat, code assistant, image generator).
Estimate which ingredient is its main competitive advantage and why.
Predict what will change first for that product: the data, the compute, or the algorithm.
Identify what signal in the wild would confirm your prediction within 12 months.
Write it down and revisit it later. You will be grading yourself on real-world feedback.

Everything in AI is ultimately a budget problem with three line items.
— A systems researcher

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-creators-three-ingredients-capstone

What is the core idea behind "The Three Ingredients: Data, Compute, Algorithms (Capstone)"?
1. Every AI breakthrough of the past decade rests on three interacting ingredients. Synthesize everything you have learned into one working model.
2. It is like teaching a smart kid your house rules.
3. Make every existing tool magically MCP-compatible — adoption takes time
4. Speed up training and inference 2-4x on modern GPUs
Which term best describes a foundational idea in "The Three Ingredients: Data, Compute, Algorithms (Capstone)"?
1. compute
2. data
3. algorithm
4. bitter lesson
A learner studying The Three Ingredients: Data, Compute, Algorithms (Capstone) would need to understand which concept?
1. data
2. algorithm
3. compute
4. bitter lesson
Which of these is directly relevant to The Three Ingredients: Data, Compute, Algorithms (Capstone)?
1. data
2. compute
3. bitter lesson
4. algorithm
Which of the following is a key point about The Three Ingredients: Data, Compute, Algorithms (Capstone)?
1. Data bottlenecks: public text ran out, driving synthetic and multimodal data
2. Compute bottlenecks: GPU supply and energy availability cap scaling
3. Algorithm bottlenecks: pure transformers plateau, pushing MoE, state space, and hybrid architectures
4. Adding one ingredient often just exposes a shortage of another
Which of these does NOT belong in a discussion of The Three Ingredients: Data, Compute, Algorithms (Capstone)?
1. Algorithm bottlenecks: pure transformers plateau, pushing MoE, state space, and hybrid architectures
2. It is like teaching a smart kid your house rules.
3. Compute bottlenecks: GPU supply and energy availability cap scaling
4. Data bottlenecks: public text ran out, driving synthetic and multimodal data
Which statement is accurate regarding The Three Ingredients: Data, Compute, Algorithms (Capstone)?
1. Choose your compute: 8 H100 GPUs for 72 hours
2. Choose your architecture: 1B-parameter decoder transformer with rotary embeddings and grouped-query …
3. Choose your data: 1B tokens of curated web + code
4. Tokenize the corpus with a 32k-token BPE vocabulary
Which of these does NOT belong in a discussion of The Three Ingredients: Data, Compute, Algorithms (Capstone)?
1. Choose your architecture: 1B-parameter decoder transformer with rotary embeddings and grouped-query …
2. Choose your compute: 8 H100 GPUs for 72 hours
3. It is like teaching a smart kid your house rules.
4. Choose your data: 1B tokens of curated web + code
What is the key insight about "Bitter lesson revisited" in the context of The Three Ingredients: Data, Compute, Algorithms (Capstone)?
1. Rich Sutton's bitter lesson says general methods plus compute beat clever methods plus human priors.
2. It is like teaching a smart kid your house rules.
3. Make every existing tool magically MCP-compatible — adoption takes time
4. Speed up training and inference 2-4x on modern GPUs
What is the recommended tip about "Ground your practice in fundamentals" in the context of The Three Ingredients: Data, Compute, Algorithms (Capstone)?
1. It is like teaching a smart kid your house rules.
2. Every AI capability has an underlying mechanism. Understanding that mechanism tells you where it'll fail — which is more…
3. Make every existing tool magically MCP-compatible — adoption takes time
4. Speed up training and inference 2-4x on modern GPUs
Which statement accurately describes an aspect of The Three Ingredients: Data, Compute, Algorithms (Capstone)?
1. It is like teaching a smart kid your house rules.
2. Make every existing tool magically MCP-compatible — adoption takes time
3. At the broadest possible level, every AI system in history is a mix of three ingredients. Data provides the examples.
4. Speed up training and inference 2-4x on modern GPUs
What does working with The Three Ingredients: Data, Compute, Algorithms (Capstone) typically involve?
1. It is like teaching a smart kid your house rules.
2. Make every existing tool magically MCP-compatible — adoption takes time
3. Speed up training and inference 2-4x on modern GPUs
4. The big idea: AI is not magic. It is three ingredients, thousands of engineers, and careful attention to which ingredient is scarce this yea…
Which best describes the scope of "The Three Ingredients: Data, Compute, Algorithms (Capstone)"?
1. It focuses on Every AI breakthrough of the past decade rests on three interacting ingredients. Synthesize everythi
2. It is unrelated to foundations workflows
3. It applies only to the opposite beginner tier
4. It was deprecated in 2024 and no longer relevant
Which section heading best belongs in a lesson about The Three Ingredients: Data, Compute, Algorithms (Capstone)?
1. It is like teaching a smart kid your house rules.
2. The ingredients mapped to history
3. Make every existing tool magically MCP-compatible — adoption takes time
4. Speed up training and inference 2-4x on modern GPUs
Which section heading best belongs in a lesson about The Three Ingredients: Data, Compute, Algorithms (Capstone)?
1. It is like teaching a smart kid your house rules.
2. Make every existing tool magically MCP-compatible — adoption takes time
3. How the ingredients interact
4. Speed up training and inference 2-4x on modern GPUs

← Back to interactive lesson