Concept · LLMs in production
Quantization
Store weights in 8 or 4 bits instead of 16, so a model fits on a smaller GPU at a small cost in quality.
3 minint4int8GGUFAWQ
A model's weights are billions of numbers, usually stored as 16-bit floats (FP16/BF16). Quantization stores them with fewer bits, such as 8 or 4, by mapping each small group of weights onto a coarser grid plus a scale factor.
memory for weights ≈ parameters × bytes per weight
8B model: 16-bit ≈ 16 GB 8-bit ≈ 8 GB 4-bit ≈ 4–5 GB
70B model: 16-bit ≈ 140 GB 8-bit ≈ 70 GB 4-bit ≈ 35–40 GBAdd memory for the KV cache, which grows with context length and the number of concurrent requests, plus runtime overhead. In practice that's often 20–50% more than the weights alone.
What you gain and lose
- Fits on cheaper hardware. A 70B model at 4-bit fits on two 24–48 GB GPUs instead of needing several 80 GB ones. An 8B model runs on a laptop.
- Often faster. Generating tokens is mostly limited by memory bandwidth, so moving fewer bytes per token helps, depending on whether the runtime has fast kernels for the format.
- Some quality loss. At 8 bits it's usually negligible. At 4 bits with a good method it's typically small on general benchmarks, but it can be noticeable on maths, code, long contexts or non-English text. Below 4 bits, quality drops quickly.
Common formats
| Format | Typical use |
|---|---|
| GGUF (Q4_K_M, Q5_K_M, Q8_0…) | llama.cpp and Ollama; CPU and Apple Silicon; local use |
| AWQ / GPTQ | 4-bit GPU serving with vLLM and similar servers |
| FP8 | Recent data-centre GPUs; close to 16-bit quality at half the memory |
Quantization can also apply to the KV cache, which lets you fit longer contexts or more concurrent users. That's separate from quantizing the weights.
Choosing
- Start with the largest model that fits in 4–5 bits rather than a smaller model at 16 bits. That usually wins on quality for the same memory.
- Evaluate on your own task, not just leaderboards. Run your evals against the 16-bit version and the quantized one.
- Use a hosted API if you don't have a reason to self-host, such as data residency, cost at very high volume, offline use, or fine-tuned weights. The self-hosting lesson covers the trade-off.