Mechanistic Interpretability: Reading the Model's Mind

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-safety-interpretability-creators

1. What is the core problem that motivated the development of sparse autoencoders?

   1. Neurons in neural networks are too small to store meaningful information
   2. Attention heads cannot be visualized effectively
   3. Models lack sufficient training data to learn consistent representations
   4. Individual neurons encode multiple unrelated concepts, making model behavior hard to interpret

2. According to the superposition hypothesis, why do most neurons in large language models appear polysemantic?

   1. Researchers mislabel neurons due to insufficient training data
   2. Polysemanticity is an inevitable result of the attention mechanism
   3. The training process intentionally mixes unrelated concepts together
   4. Models pack more features than they have dimensions, causing features to interfere with each other

3. What architectural choice allows a sparse autoencoder to produce monosemantic features?

   1. Reducing the number of attention heads to focus on key patterns
   2. Training on smaller datasets to prevent feature mixing
   3. Adding a second attention layer to refine interpretations
   4. Using a much wider hidden layer with an L1 sparsity penalty during training

4. What did Anthropic's 'Golden Gate Claude' experiment demonstrate about features?

   1. Features can only be identified in smaller models, not larger ones
   2. Features disappear when models are fine-tuned
   3. Features have a causal effect on model behavior when amplified or suppressed
   4. Features are purely correlational and cannot be manipulated

5. In the terminology of mechanistic interpretability, what is the relationship between features and circuits?

   1. Features are the nouns (concepts) while circuits are the verbs (computations that connect features)
   2. Features and circuits are the same thing described differently
   3. Circuits represent individual neurons and features represent attention patterns
   4. Features and circuits cannot be studied together in current research

6. Why might mechanistic interpretability be strategically important for AI safety compared to behavior-based evaluations alone?

   1. Interpretability research is no longer needed since models pass safety tests
   2. Behavior-based evaluations can detect all possible harmful behaviors
   3. Behavior-based evaluations are faster and cheaper than interpretability research
   4. Interpretability can distinguish genuinely aligned models from those that are deceptively aligned

7. What computational role do induction heads play in transformer models?

   1. They prevent models from repeating the same tokens
   2. They copy patterns from earlier in a sequence to predict what comes next
   3. They generate random tokens to help models maintain creativity
   4. They calculate attention scores between different layers

8. What capability does 'steering' via feature manipulation enable researchers to do?

   1. Modify the model's underlying architecture
   2. Completely replace the model's knowledge base
   3. Prevent the model from generating any text
   4. Amplify or suppress specific concepts in model outputs without retraining

9. What is one key limitation of sparse autoencoders as mentioned in the material?

   1. They may miss features with heavy L1 penalty due to the sparsity constraint
   2. They work equally well on all model architectures
   3. They perfectly reconstruct all activations without any loss
   4. They cannot be trained on activations from large models

10. Why is finding interpretable features insufficient on its own for complete model understanding?

    1. Features are model-specific and change with every update
    2. Features only tell you what concepts exist, not how they connect to produce behavior
    3. Features cannot be detected in models with more than 10 billion parameters
    4. Features are not yet discoverable in most models

11. What economic consideration was raised about scaling interpretability to frontier models?

    1. Training SAEs costs a meaningful fraction of the model's pretraining compute
    2. Interpretability research has been abandoned due to high costs
    3. Interpretability tools are free and require no computational resources
    4. Only academic institutions can afford interpretability work

12. What did Anthropic's 'Scaling Monosemanticity' paper successfully demonstrate?

    1. Features cannot be extracted from Claude models
    2. Smaller models are more interpretable than larger ones
    3. SAEs can perfectly reconstruct all model activations without loss
    4. Sparse autoencoders can extract millions of interpretable features from a frontier model

13. What does it mean that a feature is 'multilingual'?

    1. The feature only activates on text in one language
    2. The same feature fires for a concept regardless of which language expresses it
    3. The feature only activates on machine-translated content
    4. The feature translates text between languages automatically

14. What risk exists even with automated feature labeling?

    1. Automated labeling is perfectly accurate
    2. Automation always produces better labels than humans
    3. Human interpretation is still needed and can be wrong
    4. Labels cannot be assigned to features at all

15. The quote 'We have the first plausible path to actually reading the mind of a neural network' refers most directly to what development?

    1. The discovery of polysemantic neurons
    2. The invention of the transformer architecture
    3. The creation of behavior-based evaluation benchmarks
    4. Sparse autoencoders that extract interpretable features from model activations