AI Model Quantization: 4-bit, 8-bit, FP16 Tradeoffs

End-of-lesson check

15 questions · take it digitally for instant feedback at tendril.neural-forge.io/learn/quiz/end-model-families-AI-and-quantization-tradeoffs-creators

1. A developer is running a 70-billion parameter model on a single GPU with 24GB VRAM. Which quantization level would most likely allow the model to fit in memory while maintaining reasonable quality?

   1. INT8
   2. FP16
   3. FP32
   4. INT4

2. A student notices that their quantized model performs well on summarization tasks but poorly on mathematical proofs. What best explains this pattern?

   1. The quantization process introduced random errors in numerical calculations
   2. Quantization improves creative tasks but degrades factual ones
   3. The model was never trained on mathematical content
   4. Quantization quality loss affects reasoning and math tasks more than common tasks

3. What does it mean to say that quantization is 'the easiest to misconfigure' among inference optimization techniques?

   1. Quantization settings cannot be changed after deployment
   2. The tradeoffs between quality, speed, and cost are not obvious without testing
   3. It requires the most expensive hardware to implement correctly
   4. It is the only optimization technique that damages model accuracy

4. A company runs the same model at INT8 for one customer and FP16 for another. If both are serving identical requests, which customer is likely paying more per 1,000 tokens?

   1. The FP16 customer
   2. The customer with more RAM
   3. The INT8 customer
   4. They pay the same because the model is identical

5. What is the primary purpose of running task-specific evaluations across different quantization levels?

   1. To measure the actual quality delta for your particular use case
   2. To determine which quantization method was used during training
   3. To verify that the model is properly installed
   4. To compare different model architectures

6. Which statement about INT4 quantization is most accurate?

   1. It is faster than FP16 on all hardware configurations
   2. It was developed specifically for reasoning tasks
   3. It provides the greatest memory and cost savings but typically causes noticeable quality degradation
   4. It eliminates quality loss entirely by using smarter rounding

7. A developer chooses INT8 over INT4 for their production chatbot. What is the most likely reason for this choice?

   1. INT8 uses less memory than INT4
   2. INT8 was not available for their model
   3. INT8 maintains better quality for their use case
   4. INT8 requires less compute power

8. GPTQ and AWQ are mentioned in the lesson as examples of what?

   1. Cloud hosting providers
   2. Quantization methods or algorithms
   3. Evaluation benchmarks
   4. Model architectures

9. A researcher wants to benchmark a model on a coding task. They plan to use INT4 to save costs. What should they expect based on the lesson?

   1. Quality will likely degrade more on coding tasks than on general conversation
   2. INT4 will improve coding performance through better optimization
   3. The coding performance should be similar to FP16 because code is text
   4. Coding tasks are immune to quantization effects

10. Why might a startup choose to run their AI application at INT4 despite quality concerns?

    1. Lower hardware costs make the business model viable
    2. INT4 produces more accurate responses than higher precision levels
    3. INT4 is required by law for consumer applications
    4. Their users prefer faster responses over accurate ones

11. What does the lesson recommend as a 'default' quantization level for a model like Llama, Mistral, or Qwen?

    1. FP16
    2. INT8
    3. It depends on the specific task and evaluation results
    4. INT4

12. A model quantized to INT8 uses approximately what fraction of the memory compared to its FP16 version?

    1. 50%
    2. 75%
    3. 100%
    4. 25%

13. The lesson warns against promising 'zero quality loss' with quantization. Why is this unrealistic?

    1. Quantization algorithms always have bugs
    2. Some information is always lost when reducing precision from FP16 to lower bit depths
    3. Quality actually improves with quantization due to reduced noise
    4. Zero quality loss is only possible with FP32

14. When would FP16 be the preferred choice over INT8 or INT4, even though it costs more?

    1. When serving millions of concurrent users
    2. When reasoning quality is critical and hardware budget allows
    3. When the model only needs to run once
    4. When running on very old hardware with limited memory

15. What is the relationship between throughput and quantization level?

    1. Quantization always reduces throughput due to decompression overhead
    2. Quantization has no effect on throughput
    3. Higher quantization (fewer bits) generally increases throughput because less data is processed
    4. Throughput depends only on the GPU model, not quantization