Quantization fundamentals: bits, accuracy, and serving cost

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-creators-quantization-fundamentals

1. What is the main reason developers apply quantization to AI models used in production?

   1. To make the model easier to fine-tune on new data
   2. To increase the model's accuracy on benchmark datasets
   3. To add new capabilities the original model didn't have
   4. To reduce the computational resources required to run the model

2. If a model uses int8 quantization, how many bits are used to represent each weight?

   1. 32 bits
   2. 4 bits
   3. 8 bits
   4. 16 bits

3. What is a perplexity gap?

   1. The number of parameters removed during quantization
   2. The time delay when a model generates consecutive tokens
   3. The difference in training loss between epochs
   4. The difference in perplexity between a quantized model and its full-precision baseline

4. Why is it risky to assume quantization will have minimal accuracy impact based only on published research papers?

   1. Papers are always wrong about quantization effects
   2. Papers don't measure latency improvements
   3. Papers typically test on different data distributions than your production use case
   4. Papers always exaggerate the impact of quantization

5. What does post-training quantization (PTQ) refer to?

   1. Training a model from scratch using low-precision numbers
   2. Quantizing a model after it has already been trained on full-precision data
   3. Training a model specifically to be robust to quantization
   4. Adding quantization layers during the training process

6. Why might int4 quantization cause severe accuracy loss on one task but not another?

   1. The model architecture changes between tasks
   2. Some tasks require the model to store more detailed numerical information in weights
   3. Quantization software is buggy for certain tasks
   4. GPU drivers handle different tasks differently

7. What does 'serving cost' refer to in model deployment?

   1. The computational resources (memory, compute, latency) required to run the model in production
   2. The money spent on training the model
   3. The electricity used during model training
   4. The cost of labeled data for evaluation

8. You test int8 quantization and find it works well on most of your workload but causes significant errors on a specific task slice. What should you do?

   1. Revert to higher precision (fp16) for that specific task slice while using int8 elsewhere
   2. Deploy int8 everywhere since overall accuracy is acceptable
   3. Fine-tune the original model on that task
   4. Stop using quantization entirely

9. Why cannot accuracy loss from quantization be predicted without measurement on your own data?

   1. Mathematical formulas for prediction don't exist
   2. Your specific data distribution and tasks may stress different weight values than benchmarks
   3. Quantization effects are always random
   4. Measurement tools are unreliable

10. Why is end-to-end latency testing required after quantizing a model, rather than just calculating theoretical speedup?

    1. Quantization changes memory access patterns and may introduce overhead that calculations miss
    2. Calculations already account for all factors
    3. Latency testing is only needed for training, not serving
    4. Theoretical calculations are always wrong

11. What does it mean to 'revert to higher precision' during quantization evaluation?

    1. To use more aggressive quantization (e.g., from int4 to int2)
    2. To delete the quantized model entirely
    3. To train the model again from scratch
    4. To fall back to a higher precision format (e.g., from int4 to int8 or fp16) when accuracy drops below a threshold

12. When measuring quantization impact on a language model, what does perplexity specifically capture?

    1. How many parameters were reduced
    2. How confused the model is when predicting the next token (lower is better)
    3. How many tokens the model can process per second
    4. How quickly the model generates text

13. What specific aspect of model serving does int4 quantization directly reduce?

    1. The number of model parameters
    2. The amount of training data needed
    3. GPU compute operations and memory bandwidth required per inference
    4. Model training time

14. Under what condition should you keep a model at fp16 instead of quantizing to int8 or int4?

    1. When accuracy requirements are so strict that any quantization loss is unacceptable
    2. When the model is used for batch processing only
    3. When the model is very small
    4. When you have unlimited GPU memory

15. What is the primary difference between int8 and int4 quantization beyond bit depth?

    1. Int8 requires special hardware that int4 does not
    2. Int8 can only be applied to CNNs, not transformers
    3. Int4 always runs faster on every hardware
    4. Int4 typically offers larger memory and latency savings but with higher risk of accuracy degradation compared to int8