Feature Discovery in LLMs

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-safety2-feature-discovery-creators

1. In the context of LLM interpretability, what is a 'feature'?

   1. A weight matrix that controls attention between tokens
   2. A type of training data that improves model safety
   3. A specific neuron that always fires when certain keywords appear
   4. A direction in activation space whose magnitude correlates with the presence of some input property

2. What does 'activation maximization' refer to in feature discovery?

   1. Reducing the number of active neurons in a layer
   2. Comparing activations from truthful versus deceptive inputs
   3. Measuring the average activation of a neuron across a training dataset
   4. Finding inputs that make a neuron or direction fire as hard as possible

3. How do sparse autoencoders contribute to feature discovery?

   1. They maximize the sparsity of neural network weights during training
   2. They learn a dictionary of features directly from observed activations
   3. They automatically label features based on the input data
   4. They compare activations between two different model architectures

4. What is the 'probing' method for finding features?

   1. Calculating the mean activation for each neuron in a layer
   2. Inserting probe tokens into the input to observe their attention patterns
   3. Training a small classifier to detect whether a concept is present in activations
   4. Using gradient descent to maximize the difference between two activation sets

5. The 'difference in means' method for feature discovery involves:

   1. Finding inputs where activation equals zero
   2. Comparing average activations across two labeled sets of inputs
   3. Calculating the difference between forward and backward passes
   4. Computing the mean activation of each neuron individually

6. What are 'steering vectors' used for in interpretability research?

   1. Adding a specific direction to model activations to deliberately change output behavior
   2. Directing the model's attention toward particular tokens in the input
   3. Steering the model away from certain training examples
   4. Calculating the gradient of the loss function during training

7. Which research discovery identified 'induction heads' as attention patterns that implement in-context learning?

   1. Olsson et al. 2022
   2. Nanda et al. 2023
   3. Marks and Tegmark 2023
   4. Radford et al. 2019

8. What did Nanda et al. 2023 accomplish in interpretability research?

   1. Discovered steering vectors for honesty in GPT models
   2. Reverse-engineered how a small transformer performs modular addition
   3. Found that sparse autoencoders can decode all model behavior
   4. Proved that features are impossible to interpret in large models

9. Marks and Tegmark 2023 identified directions in LLM activations that correspond to:

   1. Deceptive versus honest responses
   2. English versus non-English languages
   3. Technical versus non-technical content
   4. True versus false statements

10. What are 'refusal features' in LLM interpretability?

    1. Directions that cause the model to refuse any request
    2. Weights that control whether the model can refuse to answer
    3. Activation patterns that become active just before a safety refusal is generated
    4. Features that detect when users are refusing to follow instructions

11. A key caveat about named features in interpretability is that:

    1. Named features are always accurate representations of model behavior
    2. Only features that match human concepts should be reported
    3. The name reflects the researcher's hypothesis, not necessarily the model's internal concept
    4. Features should never be named to avoid biasing results

12. 'Code-vulnerability features' in LLM research refer to:

    1. Directions that cause models to generate vulnerable code
    2. Weights that prevent models from writing code at all
    3. Activation patterns that correlate with security vulnerabilities in source code
    4. Features that detect when code is being written incorrectly

13. The lesson describes features as what in modern interpretability?

    1. The primary training objective
    2. A type of neural network architecture
    3. The fundamental unit of analysis
    4. A method for collecting training data

14. A researcher finds a feature that fires strongly on examples of deceptive statements from training data. What important caution does this illustrate?

    1. The feature should be removed from the model immediately
    2. The feature may detect 'deception example' rather than actual deception
    3. This proves the model has a theory of mind
    4. The model has definitely learned to be deceptive

15. Which feature discovery method is best described as 'optimizing input to maximize activation'?

    1. Activation maximization
    2. Sparse autoencoders
    3. Probing
    4. Difference in means