In-Context Learning

Show a model three examples, and it learns the task on the spot — without any weight updates. This is one of the strangest properties of transformers.

38 min · Reviewed 2026

Learning Without Training

In-context learning (ICL) is a model's ability to acquire a new task from examples shown in its prompt, without any gradient updates. GPT-3's 2020 paper was the breakthrough: show the model three examples of a task, and it would continue the pattern for new inputs.

Few-shot in-context learning example: English: The cat sat on the mat. Pirate: The scallywag perched on yonder rug. English: I love pizza. Pirate: Arr, me heart beats for pizza pie! English: Where is the library? Pirate: (the model continues the pattern it inferred)No fine-tuning. The model learns the 'English-to-pirate' mapping from the examples in the prompt.

Why it matters

No training required for new tasks
Adapts at runtime based on what you show
Works for tasks that did not exist when the model was trained
Lets non-ML users customize behavior with text alone

What is really happening inside

Research has shown that transformers can implement something like gradient descent in their attention layers, using the in-context examples as 'training data' for a short forward-pass 'optimization.' The model is doing a kind of learning — it just lives in the forward pass, not in the weights.

When ICL fails

Tasks far from pretraining distribution — few-shot cannot invent new capabilities
Examples in the prompt are noisy or inconsistent
Context window runs out before you can include enough examples
The model pattern-matches to a memorized template instead of the actual mapping

Our largest model, GPT-3, with 175 billion parameters, achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks.
— Brown et al., Language Models are Few-Shot Learners (2020)

The big idea: modern LLMs learn inside a single forward pass. Understanding ICL reshapes how you think about what 'training' even means.

End-of-lesson check

8 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-creators-in-context-learning

What is the main idea of "In-Context Learning"?
1. Show a model three examples, and it learns the task on the spot — without any weight updates. This is one of the strangest properties of transformers.
2. Use AI as the final authority for the whole decision
3. Avoid checking the answer once it sounds polished
4. Focus only on speed instead of judgment
Which concept is most central to "In-Context Learning"?
1. few-shot
2. in-context learning
3. meta-learning
4. GPT-3
Which use of AI fits this topic best?
1. Let the AI decide what matters without your review
2. Use the answer before checking whether it fits the situation
3. No training required for new tasks
4. Treat the AI output as automatically correct
What should a careful learner remember about "Induction heads"?
1. Use AI to draft or organize ideas about in-context learning, then verify before acting.
2. Skip the context so the tool can guess faster
3. Treat the output as private even after sharing it online
4. Use the answer without checking the source
You want to use AI after this lesson. What is the safest next step?
1. Act immediately because the AI answer is written clearly
2. Use AI for drafting and comparison, but verify before publishing or relying on it.
3. Hide uncertainty so the final answer looks cleaner
4. Use private or sensitive details before checking permission
How should AI output about in-context learning be treated?
1. As proof that no other source is needed
2. As a replacement for context, consent, or expert review
3. As a draft or helper output that still needs human judgment and verification
4. As something that becomes correct when it sounds confident
Name one way to verify an AI answer about in-context learning.
Which action would help you apply "In-Context Learning" responsibly?
1. Use the tool to avoid thinking through the tradeoff
2. Keep going even if the output conflicts with a trusted source
3. Treat the AI output as automatically correct
4. Adapts at runtime based on what you show

← Back to interactive lesson

Tendril · Creators · AI Foundations

In-Context Learning

Show a model three examples, and it learns the task on the spot — without any weight updates. This is one of the strangest properties of transformers.

38 min · Reviewed 2026

Learning Without Training

Few-shot in-context learning example: English: The cat sat on the mat. Pirate: The scallywag perched on yonder rug. English: I love pizza. Pirate: Arr, me heart beats for pizza pie! English: Where is the library? Pirate: (the model continues the pattern it inferred)No fine-tuning. The model learns the 'English-to-pirate' mapping from the examples in the prompt.

Why it matters

No training required for new tasks
Adapts at runtime based on what you show
Works for tasks that did not exist when the model was trained
Lets non-ML users customize behavior with text alone

What is really happening inside

When ICL fails

Tasks far from pretraining distribution — few-shot cannot invent new capabilities
Examples in the prompt are noisy or inconsistent
Context window runs out before you can include enough examples
The model pattern-matches to a memorized template instead of the actual mapping

Our largest model, GPT-3, with 175 billion parameters, achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks.
— Brown et al., Language Models are Few-Shot Learners (2020)

The big idea: modern LLMs learn inside a single forward pass. Understanding ICL reshapes how you think about what 'training' even means.

End-of-lesson check

8 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-creators-in-context-learning

What is the main idea of "In-Context Learning"?
1. Show a model three examples, and it learns the task on the spot — without any weight updates. This is one of the strangest properties of transformers.
2. Use AI as the final authority for the whole decision
3. Avoid checking the answer once it sounds polished
4. Focus only on speed instead of judgment
Which concept is most central to "In-Context Learning"?
1. few-shot
2. in-context learning
3. meta-learning
4. GPT-3
Which use of AI fits this topic best?
1. Let the AI decide what matters without your review
2. Use the answer before checking whether it fits the situation
3. No training required for new tasks
4. Treat the AI output as automatically correct
What should a careful learner remember about "Induction heads"?
1. Use AI to draft or organize ideas about in-context learning, then verify before acting.
2. Skip the context so the tool can guess faster
3. Treat the output as private even after sharing it online
4. Use the answer without checking the source
You want to use AI after this lesson. What is the safest next step?
1. Act immediately because the AI answer is written clearly
2. Use AI for drafting and comparison, but verify before publishing or relying on it.
3. Hide uncertainty so the final answer looks cleaner
4. Use private or sensitive details before checking permission
How should AI output about in-context learning be treated?
1. As proof that no other source is needed
2. As a replacement for context, consent, or expert review
3. As a draft or helper output that still needs human judgment and verification
4. As something that becomes correct when it sounds confident
Name one way to verify an AI answer about in-context learning.
Which action would help you apply "In-Context Learning" responsibly?
1. Use the tool to avoid thinking through the tradeoff
2. Keep going even if the output conflicts with a trusted source
3. Treat the AI output as automatically correct
4. Adapts at runtime based on what you show

← Back to interactive lesson