Feature Discovery in LLMs

A feature is a direction in activation space that corresponds to a concept. Finding them — naming them, ranking them, connecting them — is one of the central activities of interpretability research.

37 min · Reviewed 2026

What Counts as a Feature

Informally, a feature is anything the network appears to 'represent' in its activations. More formally, it's a direction in activation space whose magnitude correlates with the presence of some input property — a word, a concept, a category, a sentiment.

Ways researchers find them

Activation maximization: find inputs that make a neuron or direction fire hardest
Sparse autoencoders: learn a dictionary of features from activations
Probing: train a small classifier to detect a concept from activations
Difference in means: compare activations across two labeled sets
Steering vectors: find a direction that, when added to activations, changes output in a specific way

Named features in the wild (as of ~2024-2025)

Truthfulness and deception directions
Refusal features that activate before a safety refusal
Sycophancy features tied to agreeing with users
Entity-tracking features for names and pronouns
Code-vulnerability features
Sandbagging-related features

Every feature we name is a hypothesis. The test is whether intervening on it does what we expect.
— Neel Nanda, DeepMind

The big idea: features are the unit of analysis in modern interpretability. Learning to read them is the closest thing the field has to learning to read code.

End-of-lesson check

8 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-safety2-feature-discovery-creators

What is the main idea of "Feature Discovery in LLMs"?
1. A feature is a direction in activation space that corresponds to a concept.
2. Use AI as the final authority for the whole decision
3. Avoid checking the answer once it sounds polished
4. Focus only on speed instead of judgment
Which concept is most central to "Feature Discovery in LLMs"?
1. activation
2. feature
3. interpretability
4. probing
Which use of AI fits this topic best?
1. Let the AI decide what matters without your review
2. Use the answer before checking whether it fits the situation
3. Activation maximization: find inputs that make a neuron or direction fire hardest
4. Treat the AI output as automatically correct
What should a careful learner remember about "Landmark examples"?
1. Use AI to draft or organize ideas about feature, then verify before acting.
2. Skip the context so the tool can guess faster
3. Treat the output as private even after sharing it online
4. Use the answer without checking the source
You want to use AI after this lesson. What is the safest next step?
1. Act immediately because the AI answer is written clearly
2. AI cannot make the human values decision for you.
3. Hide uncertainty so the final answer looks cleaner
4. Use private or sensitive details before checking permission
How should AI output about feature be treated?
1. As proof that no other source is needed
2. As a replacement for context, consent, or expert review
3. As a draft or helper output that still needs human judgment and verification
4. As something that becomes correct when it sounds confident
Name one way to verify an AI answer about feature.
Which action would help you apply "Feature Discovery in LLMs" responsibly?
1. Use the tool to avoid thinking through the tradeoff
2. Keep going even if the output conflicts with a trusted source
3. Treat the AI output as automatically correct
4. Sparse autoencoders: learn a dictionary of features from activations

← Back to interactive lesson

Tendril · Creators · Ethics & Society

Feature Discovery in LLMs

37 min · Reviewed 2026

What Counts as a Feature

Ways researchers find them

Activation maximization: find inputs that make a neuron or direction fire hardest
Sparse autoencoders: learn a dictionary of features from activations
Probing: train a small classifier to detect a concept from activations
Difference in means: compare activations across two labeled sets
Steering vectors: find a direction that, when added to activations, changes output in a specific way

Named features in the wild (as of ~2024-2025)

Truthfulness and deception directions
Refusal features that activate before a safety refusal
Sycophancy features tied to agreeing with users
Entity-tracking features for names and pronouns
Code-vulnerability features
Sandbagging-related features

Every feature we name is a hypothesis. The test is whether intervening on it does what we expect.
— Neel Nanda, DeepMind

The big idea: features are the unit of analysis in modern interpretability. Learning to read them is the closest thing the field has to learning to read code.

End-of-lesson check

8 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-safety2-feature-discovery-creators

What is the main idea of "Feature Discovery in LLMs"?
1. A feature is a direction in activation space that corresponds to a concept.
2. Use AI as the final authority for the whole decision
3. Avoid checking the answer once it sounds polished
4. Focus only on speed instead of judgment
Which concept is most central to "Feature Discovery in LLMs"?
1. activation
2. feature
3. interpretability
4. probing
Which use of AI fits this topic best?
1. Let the AI decide what matters without your review
2. Use the answer before checking whether it fits the situation
3. Activation maximization: find inputs that make a neuron or direction fire hardest
4. Treat the AI output as automatically correct
What should a careful learner remember about "Landmark examples"?
1. Use AI to draft or organize ideas about feature, then verify before acting.
2. Skip the context so the tool can guess faster
3. Treat the output as private even after sharing it online
4. Use the answer without checking the source
You want to use AI after this lesson. What is the safest next step?
1. Act immediately because the AI answer is written clearly
2. AI cannot make the human values decision for you.
3. Hide uncertainty so the final answer looks cleaner
4. Use private or sensitive details before checking permission
How should AI output about feature be treated?
1. As proof that no other source is needed
2. As a replacement for context, consent, or expert review
3. As a draft or helper output that still needs human judgment and verification
4. As something that becomes correct when it sounds confident
Name one way to verify an AI answer about feature.
Which action would help you apply "Feature Discovery in LLMs" responsibly?
1. Use the tool to avoid thinking through the tradeoff
2. Keep going even if the output conflicts with a trusted source
3. Treat the AI output as automatically correct
4. Sparse autoencoders: learn a dictionary of features from activations

← Back to interactive lesson