Mechanistic Interpretability: Reading the Model's Mind

Why Opening the Box Matters

Behavior-based evaluations tell you what a model does. Interpretability tries to tell you why. If deceptive alignment is a real concern, you cannot catch it from outputs alone. You need to look inside. Mechanistic interpretability is the research program of understanding neural networks well enough to trust or distrust them for specific reasons.

The starting problem: polysemantic neurons

Early hopes that individual neurons encoded individual concepts were wrong. Most neurons are polysemantic: they activate on many unrelated concepts (curly hair, bird beaks, and political slogans in the same neuron). The superposition hypothesis (Elhage et al., 2022) explains this: models pack many more features than they have dimensions, using interference.

Sparse autoencoders: the breakthrough

A sparse autoencoder (SAE) is trained to reconstruct model activations through a much wider hidden layer with an L1 sparsity penalty. The wide layer can represent features one-per-unit. Running an SAE on model activations produces monosemantic features. Anthropic's Scaling Monosemanticity (May 2024) trained SAEs on Claude 3 Sonnet activations and extracted millions of interpretable features.

What features look like

• A feature for the Golden Gate Bridge, active on mentions, images, and abstract references
• Safety-relevant features for deception, sycophancy, bias, dangerous content
• Features for security vulnerabilities in code
• Features for specific people, places, emotions, and abstractions like betrayal or symmetry
• Multilingual features: same concept across languages fires one feature
• Multimodal features: same feature fires on text and image

Steering: feature manipulation

If a feature represents a concept, you can amplify it and observe the effect on output. Anthropic's Golden Gate Claude demo (May 2024) clamped the Golden Gate feature high and produced a model that obsessively steered every conversation toward the bridge. The proof of concept: features are causal, not just correlational.

Circuits: features connected

Features are the nouns; circuits are the verbs. A circuit is a chain of features and attention heads that implements a specific computation. Early circuits work identified induction heads (copying patterns), name movers (resolving he/she), and factual recall circuits. Transformer-circuits.pub publishes ongoing circuit analyses.

Simplified SAE training: Given activations x (dimension d) from a layer, train encoder E (d -> k where k >> d) and decoder D with loss: L = ||x - D(E(x))||^2 + lambda * ||E(x)||_1 reconstruction error + sparsity penalty Each of the k dimensions in E(x) becomes a 'feature.' With k in the millions and lambda tuned, features often become monosemantic.The SAE objective: reconstruct activations through a wide, sparse hidden layer.

Why this may be strategically central

• Distinguishes aligned behavior from deceptively-aligned behavior (in principle)
• Gives auditors something concrete to audit, not just outputs
• Enables targeted edits: remove dangerous capabilities without retraining
• Provides early warning for capability shifts (new features emerging)
• Creates a path to verifiable safety claims rather than confidence-based ones

Known limits

• SAEs reconstruct activations imperfectly, missing features with heavy L1
• Features are model-specific; not yet transferable across model generations
• Feature labels rely on human interpretation, which can be wrong
• Millions of features mean automation of labeling; automation has errors
• We do not yet know if all mesa-objective representations will be interpretable

We have the first plausible path to actually reading the mind of a neural network. That was not true five years ago.

— Chris Olah, Anthropic interpretability team

The big idea: for the first time, there is a plausible technical path to reading what a neural network is actually computing. It is incomplete, expensive, and young. It is also the most direct response to the harder alignment questions, and it is advancing quickly.