Grokking: Learning That Snaps Into Place

Sometimes a network memorizes, then — long after you would have stopped training — suddenly generalizes. That is grokking, a real and weird phenomenon. Why it matters beyond the toy Grokking suggests that 'more training' can sometimes qualitatively change a model's behavior — not just improve a score but switch to a different algorithm internally.

40 min · Reviewed 2026

A Weird Training Curve

In 2022, Power, Burda, Edwards, and colleagues at OpenAI reported something strange: a small transformer trained on modular arithmetic would reach ~100% training accuracy while test accuracy stayed near zero for thousands of epochs. Then, suddenly, test accuracy would snap to ~100%. They called the phenomenon grokking, after the Heinlein word for 'to understand fully.'

The classic curve

Accuracy 1.0 | Train ______________________ | / | / Test 0.5 | / ________ | / / 0.0 |____/__________/______________ Time memorize generalize (early) (much later)Training accuracy saturates. Test accuracy stays low — then snaps up far later.

What is actually happening

Early: the model memorizes training examples, complex internal representations
Later: weight decay slowly pressures the model toward simpler solutions
At the transition: an algorithmic circuit forms that actually generalizes
After: performance generalizes, internal representations simplify

Why it matters beyond the toy

Grokking suggests that 'more training' can sometimes qualitatively change a model's behavior — not just improve a score but switch to a different algorithm internally. That has implications for how we evaluate safety during training.

Scores can plateau, then jump, without any architectural change
Early-stopped models can be qualitatively different from fully-trained ones
Training dynamics (not just final weights) matter for capability forecasting
Mechanistic interpretability is the tool that catches these transitions

We show that neural networks can 'grok' algorithmic tasks, generalizing well after overfitting the training set.
— Power et al., Grokking paper (2022)

The big idea: learning is not monotonic. Grokking proves that long after training looks 'done,' the internal algorithm can still be changing.

End-of-lesson check

8 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-creators-grokking

What is the main idea of "Grokking: Learning That Snaps Into Place"?
1. Sometimes a network memorizes, then — long after you would have stopped training — suddenly generalizes.
2. Use AI as the final authority for the whole decision
3. Avoid checking the answer once it sounds polished
4. Focus only on speed instead of judgment
Which concept is most central to "Grokking: Learning That Snaps Into Place"?
1. generalization
2. grokking
3. double descent
4. phase transition
Which use of AI fits this topic best?
1. Let the AI decide what matters without your review
2. Use the answer before checking whether it fits the situation
3. Early: the model memorizes training examples, complex internal representations
4. Treat the AI output as automatically correct
What should a careful learner remember about "Mechanistic interpretability found the circuit"?
1. Use AI to draft or organize ideas about grokking, then verify before acting.
2. Skip the context so the tool can guess faster
3. Treat the output as private even after sharing it online
4. Use the answer without checking the source
You want to use AI after this lesson. What is the safest next step?
1. Act immediately because the AI answer is written clearly
2. Use AI for drafting and comparison, but verify before publishing or relying on it.
3. Hide uncertainty so the final answer looks cleaner
4. Use private or sensitive details before checking permission
How should AI output about grokking be treated?
1. As proof that no other source is needed
2. As a replacement for context, consent, or expert review
3. As a draft or helper output that still needs human judgment and verification
4. As something that becomes correct when it sounds confident
Name one way to verify an AI answer about grokking.
Which action would help you apply "Grokking: Learning That Snaps Into Place" responsibly?
1. Use the tool to avoid thinking through the tradeoff
2. Keep going even if the output conflicts with a trusted source
3. Treat the AI output as automatically correct
4. Later: weight decay slowly pressures the model toward simpler solutions

← Back to interactive lesson

Tendril · Creators · AI Foundations

Grokking: Learning That Snaps Into Place

40 min · Reviewed 2026

A Weird Training Curve

The classic curve

Accuracy 1.0 | Train ______________________ | / | / Test 0.5 | / ________ | / / 0.0 |____/__________/______________ Time memorize generalize (early) (much later)Training accuracy saturates. Test accuracy stays low — then snaps up far later.

What is actually happening

Early: the model memorizes training examples, complex internal representations
Later: weight decay slowly pressures the model toward simpler solutions
At the transition: an algorithmic circuit forms that actually generalizes
After: performance generalizes, internal representations simplify

Why it matters beyond the toy

Scores can plateau, then jump, without any architectural change
Early-stopped models can be qualitatively different from fully-trained ones
Training dynamics (not just final weights) matter for capability forecasting
Mechanistic interpretability is the tool that catches these transitions

We show that neural networks can 'grok' algorithmic tasks, generalizing well after overfitting the training set.
— Power et al., Grokking paper (2022)

The big idea: learning is not monotonic. Grokking proves that long after training looks 'done,' the internal algorithm can still be changing.

End-of-lesson check

8 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-creators-grokking

What is the main idea of "Grokking: Learning That Snaps Into Place"?
1. Sometimes a network memorizes, then — long after you would have stopped training — suddenly generalizes.
2. Use AI as the final authority for the whole decision
3. Avoid checking the answer once it sounds polished
4. Focus only on speed instead of judgment
Which concept is most central to "Grokking: Learning That Snaps Into Place"?
1. generalization
2. grokking
3. double descent
4. phase transition
Which use of AI fits this topic best?
1. Let the AI decide what matters without your review
2. Use the answer before checking whether it fits the situation
3. Early: the model memorizes training examples, complex internal representations
4. Treat the AI output as automatically correct
What should a careful learner remember about "Mechanistic interpretability found the circuit"?
1. Use AI to draft or organize ideas about grokking, then verify before acting.
2. Skip the context so the tool can guess faster
3. Treat the output as private even after sharing it online
4. Use the answer without checking the source
You want to use AI after this lesson. What is the safest next step?
1. Act immediately because the AI answer is written clearly
2. Use AI for drafting and comparison, but verify before publishing or relying on it.
3. Hide uncertainty so the final answer looks cleaner
4. Use private or sensitive details before checking permission
How should AI output about grokking be treated?
1. As proof that no other source is needed
2. As a replacement for context, consent, or expert review
3. As a draft or helper output that still needs human judgment and verification
4. As something that becomes correct when it sounds confident
Name one way to verify an AI answer about grokking.
Which action would help you apply "Grokking: Learning That Snaps Into Place" responsibly?
1. Use the tool to avoid thinking through the tradeoff
2. Keep going even if the output conflicts with a trusted source
3. Treat the AI output as automatically correct
4. Later: weight decay slowly pressures the model toward simpler solutions

← Back to interactive lesson