Quantization Explained: GGUF, AWQ, GPTQ, and the Q4 vs Q8 vs FP16 Decision

An LLM's weights are originally floating-point numbers — typically FP16 or BF16. Quantization replaces those with lower-precision integers (often 4 or 8 bits per weight). The model gets smaller and faster. Quality drops a little. The question is always: how little, and is the drop worth the savings?

The format zoo: GGUF, AWQ, GPTQ

• GGUF: the format llama.cpp (and therefore Ollama, LM Studio) uses. CPU-and-GPU friendly, the local-model default
• AWQ: Activation-aware Weight Quantization. Common for GPU inference servers like vLLM. Good 4-bit quality
• GPTQ: an older but still common GPU-targeted quantization. Often on Hugging Face for Linux/CUDA workflows
• Native FP16 / BF16: the unquantized weights. Reference quality, large size, GPU only

What the suffixes like Q4_K_M actually mean

GGUF quants come with cryptic suffixes (Q4_K_M, Q5_K_S, Q6_K). The number is the bit width. 'K' means k-quants, a smarter quantization scheme than the original. 'S', 'M', 'L' are size variants — small, medium, large — that trade a tiny bit more space for a bit more quality. Q4_K_M is the most common 'good default' you will see in the wild.

Measuring quality, not vibes

1. 1Run llama-perplexity (in llama.cpp) on a representative text sample at FP16 vs your candidate quant
2. 2Run a short eval set of your real prompts and compare answers manually
3. 3Plot tokens-per-second on your hardware at each quant — the speed gain may surprise you
4. 4Pick the highest quant that fits your memory comfortably with your target context size

Apply this

• Pick a 7B and download both Q4_K_M and Q8 versions of the same model
• Compare answers on five representative prompts side by side
• Compare tokens-per-second on your hardware and decide which one wins for your workload

The big idea: quantization is not a tax — it is a slider. Find the highest setting your hardware allows, and only drop further when you have to.