Mixture of Depths: How AI Models Spend Compute Per Token

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-foundations-ai-mixture-of-depths-r8a4-creators

1. In a mixture-of-depths architecture, what determines whether a particular token traverses all transformer layers or skips some?

   1. A dedicated router network that learns to predict which layers are unnecessary for each token
   2. A fixed pattern that alternates between active and skipped layers
   3. The total compute budget allocated before inference begins
   4. The length of the input sequence measured in tokens

2. Why do researchers recommend including p95 and p99 latency metrics when evaluating mixture-of-depths efficiency claims?

   1. They measure the total energy consumption of the model
   2. Average latency can hide cases where certain tokens require significantly more compute, creating tail-latency surprises
   3. These metrics are required by academic publication standards
   4. Average latency alone already captures all relevant performance characteristics

3. What is a degenerate mixture-of-depths model and how does it occur?

   1. A model that only processes the first token in each sequence
   2. A model that fails to generate coherent output
   3. A model where the router consistently skips either almost no layers or almost every layer, losing the efficiency benefits
   4. A model with fewer parameters than its dense counterpart

4. Mixture-of-depths can be combined with mixture-of-experts (MoE) to achieve additional efficiency gains. What characteristic do both techniques share?

   1. Both can only be applied to the attention mechanism, not feed-forward networks
   2. Both require doubling the number of parameters in the model
   3. Both eliminate the need for any routing mechanisms
   4. Both selectively activate only a portion of the model's total capacity for each input

5. On which types of tokens does a mixture-of-depths router typically concentrate compute?

   1. Tokens that are identical to previously processed tokens
   2. Tokens that appear at the beginning of every input sequence
   3. Tokens where the router has high uncertainty about which layers to apply
   4. Tokens containing numerical data only

6. What limitation of mixture-of-depths is most relevant when deploying models in real-time applications with strict latency deadlines?

   1. The inability to process more than 512 tokens in a single sequence
   2. The need for larger memory buffers to store routing decisions
   3. The requirement for specialized hardware that most servers lack
   4. The difficulty in guaranteeing consistent latency due to variable layer skipping per token

7. What training signal is recommended as a first-class metric when training a mixture-of-depths router?

   1. The temperature setting used during sampling
   2. The number of tokens processed per second
   3. The total loss of the model during backpropagation
   4. Routing entropy, which measures the unpredictability of the router's layer selections

8. Which statement accurately describes what mixture-of-depths cannot reliably achieve, even when working correctly?

   1. Replacing the need for any model training data
   2. Matching dense-model quality on adversarial inputs drawn from the tail of the distribution
   3. Eliminating all engineering complexity from the serving infrastructure
   4. Processing all tokens with exactly the same number of FLOPs

9. What engineering challenge is introduced specifically by adding mixture-of-depths to a serving system?

   1. Reduced compatibility with existing model serialization formats
   2. Increased difficulty in batching multiple requests together
   3. Additional complexity in managing variable-length per-token computation paths
   4. The need to store routing decisions for regulatory compliance

10. A researcher claims their mixture-of-depths model uses 40% less compute on average while maintaining equal quality to a dense model. What additional information would you need to critically evaluate this claim?

    1. The programming language used to implement the router
    2. The number of tokens in their test vocabulary
    3. The exact color scheme used in their demo videos
    4. Latency distributions including p95 and p99 values to check for tail-latency issues

11. What would monitoring a router's routing entropy reveal about the model's behavior?

    1. The exact energy consumption of each inference request
    2. The speed of token generation during inference
    3. Whether the router is making diverse layer-skipping decisions or has become degenerate
    4. The total number of parameters in the model

12. In mixture-of-depths, what distinguishes 'easy' tokens from 'difficult' tokens from the router's perspective?

    1. The alphabetical position of the first letter in the token
    2. The absolute length of the word or subword in the token
    3. The presence or absence of punctuation in the token
    4. The router's learned ability to predict which tokens can be processed with fewer layers without quality loss

13. What is the primary efficiency metric that mixture-of-depths aims to reduce?

    1. The latency of the very first token in any sequence
    2. The size of the training dataset required
    3. The total number of model parameters
    4. Average compute per token while preserving downstream output quality

14. What happens when a mixture-of-depths router encounters inputs from the tail of the distribution (rare or adversarial cases)?

    1. The model automatically switches to a different architecture
    2. The router automatically routes all such tokens through every layer
    3. The router may make poor layer-skipping decisions, potentially reducing quality compared to a dense model
    4. The router crashes and returns an error message

15. Which architectural concept is most closely related to mixture-of-depths?

    1. Convolutional neural networks that use sliding window operations
    2. Conditional compute, where computation is dynamically allocated based on input characteristics
    3. Long short-term memory networks with gating mechanisms
    4. Recurrent neural networks that process sequences one element at a time