Distillation: Making Big Models Cheap

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-ai-foundations-distillation-final1-creators

1. What is the primary goal of model distillation?

   1. To make a large model produce more accurate outputs
   2. To replace frontier models entirely with smaller alternatives
   3. To compress a large model's behavior into a smaller, cheaper model
   4. To increase the training data available for machine learning

2. In teacher-student distillation, what is the 'teacher'?

   1. The smaller model that learns from the student
   2. A human expert who labels training data
   3. A large, capable model whose behavior gets replicated
   4. A dataset containing optimal input-output pairs

3. A company wants to reduce their AI API costs by 10x for a narrow, repetitive task. What approach would likely achieve this?

   1. Switch to a rule-based system with hardcoded responses
   2. Use a more powerful frontier model with more parameters
   3. Apply distillation using their current model as the teacher
   4. Deploy the same model on more expensive hardware

4. According to the recommended distillation process, how many input/output pairs should you generate from the frontier model?

   1. 100 to 500 pairs
   2. As few as 10 pairs, but quality matters more than quantity
   3. 1000 to 5000 pairs
   4. 10,000 to 50,000 pairs

5. A distilled model performs poorly on a task slightly different from its training data. What term describes this limitation?

   1. Model drift
   2. Distribution shift
   3. Brittleness outside training distribution
   4. Overfitting to the teacher

6. Why might you need to perform distillation again after initially creating a student model?

   1. The teacher model has been updated with new capabilities
   2. The hardware has become faster
   3. The student model has become too large
   4. The student model has memorized all possible inputs

7. What is a key advantage of running a distilled model on-device rather than calling a remote API?

   1. Access to more training data
   2. Higher accuracy due to more computational power
   3. Ability to use larger models than cloud services allow
   4. Eliminated network latency and reduced per-call costs

8. Which of the following is NOT a typical benefit of model distillation?

   1. 80-95% of teacher quality retention
   2. Complete replacement of the frontier model
   3. 5-20x reduction in per-call cost
   4. Running on cheaper hardware or on-device

9. What type of task is distillation LEAST suitable for?

   1. A simple sentiment analysis task
   2. A structured data extraction task
   3. An open-ended creative writing task
   4. A narrow, repetitive classification task

10. What quality level can a well-distilled model typically achieve compared to its teacher?

    1. 10-30% of teacher quality
    2. 80-95% of teacher quality
    3. 100% or more of teacher quality
    4. 40-50% of teacher quality

11. After creating a distilled model, why should the original teacher model remain available?

    1. To serve as a backup if the student model fails
    2. To verify the student model's outputs are correct
    3. To continuously train the student model in real-time
    4. To handle the long tail of tasks the distilled model can't cover

12. In the distillation workflow, what is the purpose of fine-tuning the smaller model?

    1. To reduce the model's size further after training
    2. To add new capabilities the teacher doesn't have
    3. To make the model larger and more capable
    4. To adapt the pre-trained model to replicate the teacher's behavior on specific pairs

13. What does it mean that a distilled model is 'brittle outside its training distribution'?

    1. The model fails when given inputs significantly different from its training examples
    2. The model requires constant retraining to maintain performance
    3. The model becomes less accurate over time
    4. The model will eventually forget its training

14. What type of hardware can typically run a distilled model?

    1. Hardware that requires constant internet connectivity
    2. Cheaper hardware or even consumer devices
    3. Only expensive data center GPUs
    4. Only specialized AI accelerators costing thousands of dollars

15. How does distillation affect latency compared to calling a frontier model?

    1. Latency becomes unpredictable
    2. Latency increases significantly
    3. Latency decreases substantially
    4. Latency remains the same