In-Context Learning

In-context learning (ICL) is a model's ability to acquire a new task from examples shown in its prompt, without any gradient updates. GPT-3's 2020 paper was the breakthrough: show the model three examples of a task, and it would continue the pattern for new inputs.

Few-shot in-context learning example:

English: The cat sat on the mat.
Pirate:  The scallywag perched on yonder rug.

English: I love pizza.
Pirate:  Arr, me heart beats for pizza pie!

English: Where is the library?
Pirate:

(the model continues the pattern it inferred)

Why it matters

• No training required for new tasks
• Adapts at runtime based on what you show
• Works for tasks that did not exist when the model was trained
• Lets non-ML users customize behavior with text alone

What is really happening inside

Research has shown that transformers can implement something like gradient descent in their attention layers, using the in-context examples as 'training data' for a short forward-pass 'optimization.' The model is doing a kind of learning — it just lives in the forward pass, not in the weights.

When ICL fails

• Tasks far from pretraining distribution — few-shot cannot invent new capabilities
• Examples in the prompt are noisy or inconsistent
• Context window runs out before you can include enough examples
• The model pattern-matches to a memorized template instead of the actual mapping

“Our largest model, GPT-3, with 175 billion parameters, achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks.”

The big idea: modern LLMs learn inside a single forward pass. Understanding ICL reshapes how you think about what 'training' even means.