# Quantization

> Store weights in 8 or 4 bits instead of 16, so a model fits on a smaller GPU at a small cost in quality.

Concept · LLMs in production · https://levelcode.org/concepts/quantization

A model's weights are billions of numbers, usually stored as 16-bit floats (FP16/BF16). **Quantization** stores them with fewer bits, such as 8 or 4, by mapping each small group of weights onto a coarser grid plus a scale factor.

```text
memory for weights ≈ parameters × bytes per weight

8B model:   16-bit ≈ 16 GB    8-bit ≈ 8 GB    4-bit ≈ 4–5 GB
70B model:  16-bit ≈ 140 GB   8-bit ≈ 70 GB   4-bit ≈ 35–40 GB
```

Add memory for the KV cache, which grows with [context length](https://levelcode.org/concepts/context-windows) and the number of concurrent requests, plus runtime overhead. In practice that's often 20–50% more than the weights alone.

## What you gain and lose

- **Fits on cheaper hardware.** A 70B model at 4-bit fits on two 24–48 GB GPUs instead of needing several 80 GB ones. An 8B model runs on a laptop.
- **Often faster.** Generating tokens is mostly limited by memory bandwidth, so moving fewer bytes per token helps, depending on whether the runtime has fast kernels for the format.
- **Some quality loss.** At 8 bits it's usually negligible. At 4 bits with a good method it's typically small on general benchmarks, but it can be noticeable on maths, code, long contexts or non-English text. Below 4 bits, quality drops quickly.

## Common formats

| Format | Typical use |
|---|---|
| **GGUF** (Q4_K_M, Q5_K_M, Q8_0…) | llama.cpp and Ollama; CPU and Apple Silicon; local use |
| **AWQ / GPTQ** | 4-bit GPU serving with vLLM and similar servers |
| **FP8** | Recent data-centre GPUs; close to 16-bit quality at half the memory |

Quantization can also apply to the **KV cache**, which lets you fit longer contexts or more concurrent users. That's separate from quantizing the weights.

## Choosing

- **Start with the largest model that fits in 4–5 bits** rather than a smaller model at 16 bits. That usually wins on quality for the same memory.
- **Evaluate on your own task,** not just leaderboards. Run your [evals](https://levelcode.org/concepts/llm-as-judge) against the 16-bit version and the quantized one.
- **Use a hosted API** if you don't have a reason to self-host, such as data residency, cost at very high volume, offline use, or fine-tuned weights. The [self-hosting lesson](https://levelcode.org/roadmaps/ai-engineer/self-hosting-models) covers the trade-off.