Logit Lens: Peeking at Predictions Mid-Forward-Pass

Transformers build up predictions layer by layer. Each layer reads the residual stream — a running hidden state — and writes a correction. The logit lens technique, popularized by a 2020 LessWrong post by nostalgebraist, is to apply the model's final unembedding matrix to intermediate layers, as if prediction happened there.

What you see

• Early layers: predictions close to the input token or simple patterns
• Middle layers: predictions related to the general category or topic
• Later layers: predictions refine toward the correct next token
• Near the end: the final answer crystallizes

Variants and refinements

1. 1Tuned lens (Belrose et al. 2023): train small translators per layer for more accurate reading
2. 2Logit difference: compare predictions for target vs. distractor tokens
3. 3Direct logit attribution: decompose which components contributed to a prediction
4. 4Patchscopes: use a stronger model to interpret activations from a weaker one

The big idea: the logit lens is one of the cheapest interpretability tools. One line of code gives you a new window into how a transformer thinks.