Model distillation fundamentals: smaller, faster, mostly as good

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-creators-distillation-fundamentals

1. What does it mean that the capability gap between teacher and student is 'rarely uniform across tasks'?

   1. The student model loses equal performance on every possible task
   2. Both models perform identically on simple tasks but differently on complex ones
   3. The gap can be eliminated by training longer
   4. The performance difference varies depending on which task is being evaluated

2. Why might relying solely on aggregate performance metrics be misleading when evaluating a distilled model?

   1. The student might perform well overall while failing badly on rare but high-stakes cases
   2. Aggregate metrics are always accurate and never misleading
   3. Aggregates require more computational resources to calculate
   4. Teachers rarely report aggregate metrics to students

3. What is the fundamental trade-off when using model distillation?

   1. Better privacy versus reduced performance
   2. Smaller model size versus some loss of capability compared to the teacher
   3. Faster training time versus lower energy consumption
   4. Higher accuracy versus simpler code

4. In a distillation pipeline, what data does the student model use for training?

   1. Randomly generated synthetic data
   2. The teacher model's outputs combined with input data
   3. Original human-labeled training data only
   4. A combination of the student's own predictions and human feedback

5. What does the term 'knowledge transfer' refer to in model distillation?

   1. The process of the student model learning to replicate the teacher model's behavior
   2. Converting explicit knowledge into implicit knowledge
   3. Moving a trained model from one server to another
   4. Exporting model weights to a different format

6. When designing an audit to compare teacher and student models, what is the purpose of defining 'acceptable gap thresholds'?

   1. To calculate the maximum training time allowed
   2. To determine how much the student can deviate from the teacher's pricing
   3. To set performance limits that trigger alerts when exceeded
   4. To measure the physical size difference between models

7. Why is it important to 'slice' your evaluations when assessing a distilled model?

   1. To reduce the total amount of data needed for testing
   2. To examine performance on specific subsets rather than just overall metrics
   3. To remove outlier results from the dataset
   4. To compare only the simplest test cases

8. What is a key reason a distilled student model cannot guarantee matching the teacher on rare or hardest cases?

   1. The training algorithm prevents learning from rare examples
   2. Student models are too small to handle any edge cases
   3. Rare cases often require knowledge the teacher itself may not have fully learned, making them harder to transfer
   4. Teachers deliberately withhold knowledge from students

9. What does the phrase 'mostly as good' in the lesson title imply about distilled models?

   1. They perform adequately on most tasks but with known limitations
   2. They require no further testing after distillation
   3. They should never be deployed in production
   4. They are completely identical to their teacher models

10. What is the primary motivation for deploying a distilled (student) model instead of the original teacher model?

    1. To achieve faster inference and lower computational costs
    2. To eliminate all potential biases
    3. To improve the model's reasoning capabilities
    4. To increase the amount of training data available

11. When creating an audit comparing teacher and student models, what does 'sample sizes' refer to?

    1. The total cost of training each model
    2. The file size of each model in megabytes
    3. The number of test examples evaluated per task category
    4. The amount of memory required to run each model

12. Why should task categories where distillation typically suffers most be flagged during evaluation?

    1. Because they prove the distillation process failed completely
    2. Because they require hiring additional engineers
    3. Because they indicate where the student may be unreliable and need monitoring or fallback
    4. Because these categories should be removed from the model entirely

13. What does it mean to 'estimate per-task capability gap'?

    1. Measure the performance difference between teacher and student on each individual task type
    2. Calculate the total number of parameters in both models
    3. Calculate the monetary cost difference between models
    4. Estimate the training time required for both models

14. In distillation, what is lost when compressing a large teacher model into a smaller student model?

    1. The model's ability to be updated
    2. The entire model architecture and design
    3. All ability to process text inputs
    4. Some capability, particularly on difficult or rare cases

15. What is the purpose of sketching a distillation pipeline from teacher outputs to student training?

    1. To determine the optimal number of teachers per student
    2. To calculate the exact memory requirements of each model
    3. To create visual diagrams for marketing materials
    4. To design the systematic flow of how knowledge transfers from teacher to student