# Self-hosting open models

> Run open-weight models with Ollama and vLLM, size GPU memory, choose quantization, and decide when self-hosting beats a hosted API.

AI Engineer roadmap · Agents & production · 20 min read · https://levelcode.org/roadmaps/ai-engineer/self-hosting-models

Hosted APIs are the fastest way to ship, but sometimes you need the model on hardware you control: data that may not leave your network, an offline device, a fine-tuned model, or volume high enough that renting GPUs is cheaper than paying per token. Open-weight models (Llama, Qwen, Mistral, Gemma and others) make that possible. This lesson covers when self-hosting is worth it, running models locally with Ollama and serving them with vLLM, how quantization works, how to size GPU memory with real arithmetic, the throughput-versus-latency trade-off, and what it takes to run your own model server in production. (The alternative path in this roadmap covers running hosted APIs in production instead.)

## When self-hosting is worth it

| | Hosted API | Self-hosted open model |
| --- | --- | --- |
| Quality | Frontier models | Good, but usually behind the best hosted models on hard tasks |
| Cost model | Per token; zero when idle | Per GPU-hour, whether busy or idle |
| Data location | Provider's infrastructure | Your network, even air-gapped |
| Operations | None | GPUs, drivers, scaling, upgrades, on-call |
| Customisation | Prompting | Fine-tuned weights, full control of the stack |

Do the cost arithmetic before you decide. Suppose you rent one 80 GB data-centre GPU for an assumed $2.50 an hour (prices vary widely by provider and commitment). Running all month costs about $1,825. A typical request in your app uses 2,000 input and 300 output tokens:

```text
Hosted small model, $0.10 / $0.50 per million tokens:
  2,000 x 0.10/1M + 300 x 0.50/1M = $0.00035 per request
  $1,825 buys ~5.2 million requests a month (about 2 requests/second, around the clock)

Hosted mid-tier model, $2 / $10 per million tokens:
  2,000 x 2/1M + 300 x 10/1M = $0.007 per request
  $1,825 buys ~260,000 requests a month (about 0.1 requests/second)
```

So the GPU only saves money if you have steady, high volume *and* an open model that passes your evaluation as well as the hosted model it replaces, and the comparison ignores the engineering time to run it. Cost alone rarely justifies self-hosting at small scale. Data control, offline operation and custom models more often do.

## Running models locally with Ollama

Ollama is the easiest way to run a model on a laptop or a single server. It downloads quantized models, runs them on a GPU if you have one (or Apple Silicon, or CPU), and exposes an HTTP API.

```bash
curl -fsSL https://ollama.com/install.sh | sh   # Linux; on macOS and Windows install the app
ollama pull llama3.1:8b                          # download a model (about 5 GB, 4-bit)
ollama run llama3.1:8b "Explain a B-tree index in two sentences."
ollama list                                      # models on disk
ollama ps                                        # models loaded in memory, and on GPU or CPU
```

The server listens on `http://localhost:11434`. Its native chat endpoint:

```python
import requests

response = requests.post(
    "http://localhost:11434/api/chat",
    json={
        "model": "llama3.1:8b",
        "messages": [
            {"role": "system", "content": "You are a concise assistant."},
            {"role": "user", "content": "What is a vector index?"},
        ],
        "stream": False,
        "options": {"num_ctx": 8192},  # context window; the default is small on most GPUs
    },
    timeout=120,
)
response.raise_for_status()
data = response.json()
print(data["message"]["content"])
print(f"{data['eval_count']} tokens at {data['eval_count'] / (data['eval_duration'] / 1e9):.1f} tokens/s")
```

The default context window depends on VRAM: only 4,096 tokens on GPUs with less than 24 GB (32K and 256K on larger ones); prompts longer than `num_ctx` are silently truncated, a common cause of "the model ignored my instructions" with RAG prompts. Raising it costs memory, as the sizing section explains.

> [!WARNING]
> Ollama has no authentication. It binds to localhost by default; if you change `OLLAMA_HOST` to listen on other interfaces, anyone who can reach the port can use your GPU. Put it behind a reverse proxy with authentication, or keep it on a private network.

## Serving with vLLM

Ollama is built for convenience and single users. When many users share a GPU server, use an inference engine designed for throughput. **vLLM** is the most widely used: it batches concurrent requests continuously (new requests join the running batch at each step instead of waiting for it to finish) and manages the attention cache in small pages (PagedAttention), so far more requests fit in memory at once.

On a Linux machine with an NVIDIA GPU:

```bash
pip install vllm
vllm serve Qwen/Qwen2.5-7B-Instruct \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --api-key "$VLLM_API_KEY"
```

This downloads the weights from Hugging Face (gated models such as Llama require accepting the licence and setting `HF_TOKEN`) and starts an **OpenAI-compatible** server on port 8000. Key flags:

- `--max-model-len`: the longest prompt plus output a request may use. Lower values leave room for more concurrent requests.
- `--gpu-memory-utilization`: the fraction of GPU memory vLLM claims up front (default 0.9) for weights plus attention cache.
- `--tensor-parallel-size 2`: split a model too big for one GPU across two.
- `--max-num-seqs`: cap on concurrently running requests.
- `--api-key`: require `Authorization: Bearer <key>` on requests.

Call it with plain HTTP:

```bash
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $VLLM_API_KEY" \
  -d '{
    "model": "Qwen/Qwen2.5-7B-Instruct",
    "messages": [{"role": "user", "content": "Give me three names for a hiking app."}],
    "max_tokens": 200
  }'
```

```json
{
  "id": "chatcmpl-...",
  "object": "chat.completion",
  "model": "Qwen/Qwen2.5-7B-Instruct",
  "choices": [
    {
      "index": 0,
      "message": {"role": "assistant", "content": "1. TrailMate\n2. Summit\n3. Pathfinder"},
      "finish_reason": "stop"
    }
  ],
  "usage": {"prompt_tokens": 18, "completion_tokens": 14, "total_tokens": 32}
}
```

Ollama serves the same format at `http://localhost:11434/v1/chat/completions`, so one small client function works against both, and against many other servers:

```python
import os
import requests

def chat(base_url: str, model: str, messages: list[dict], max_tokens: int = 500) -> str:
    response = requests.post(
        f"{base_url}/v1/chat/completions",
        headers={"Authorization": f"Bearer {os.environ.get('LLM_API_KEY', 'none')}"},
        json={"model": model, "messages": messages, "max_tokens": max_tokens},
        timeout=60,
    )
    response.raise_for_status()
    choice = response.json()["choices"][0]
    if choice["finish_reason"] == "length":
        raise RuntimeError("output truncated at max_tokens")
    return choice["message"]["content"]

print(chat("http://localhost:11434", "llama3.1:8b", [{"role": "user", "content": "Hi!"}]))
```

## Quantization

A model's weights are numbers. Stored in 16-bit floating point (FP16 or BF16, the usual release format), each parameter takes 2 bytes, so an 8-billion-parameter model needs 16 GB just for weights. **Quantization** stores weights with fewer bits, plus small scale factors that map them back to the original range:

| Precision | Bytes per parameter | 8B model weights | Quality |
| --- | --- | --- | --- |
| BF16 / FP16 | 2 | ~16 GB | Reference |
| 8-bit (FP8, INT8, GGUF Q8_0) | ~1 | ~8 GB | Practically identical |
| 4-bit (GGUF Q4_K_M, AWQ, GPTQ) | ~0.55-0.6 | ~4.5-5 GB | Small loss; usually fine for chat, RAG, extraction |
| 3-bit and below | <0.45 | <3.5 GB | Noticeable degradation |

Real 4-bit formats average a little over 4 bits per weight once scales are included (Q4_K_M is about 4.8). The formats you will meet:

- **GGUF**: the file format of llama.cpp, used by Ollama. Runs on CPU, Apple Silicon and GPUs. Names like `Q4_K_M` (4-bit, medium) and `Q8_0` (8-bit) describe the scheme. Ollama's default model tags are typically 4-bit.
- **AWQ and GPTQ**: 4-bit formats for GPU serving with vLLM. Download a pre-quantized repository, for example `vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ`, and vLLM detects the format.
- **FP8**: 8-bit floating point supported natively by recent NVIDIA data-centre GPUs; `--quantization fp8` in vLLM.

A useful rule: a larger model at 4-bit usually beats a smaller model at 16-bit for the same memory. But quantization hurts some tasks (precise maths, code, long-context recall) more than others, so run your own eval on the quantized model before deploying it.

## Sizing GPU memory

GPU memory has to hold three things:

1. **Weights**: parameters x bytes per parameter.
2. **KV cache**: for every token in every active request, the attention keys and values at every layer. Per token: `2 x layers x kv_heads x head_dim x bytes per value`.
3. **Overhead**: activations, CUDA context, fragmentation. Budget around 20%.

Worked example: **Llama 3.1 8B** has 32 layers, 8 key-value heads and a head dimension of 128 (from the model's `config.json`). With a 16-bit KV cache:

```text
KV per token = 2 x 32 layers x 8 heads x 128 dims x 2 bytes = 131,072 bytes = 128 KiB
One 8,192-token request = 8,192 x 128 KiB = 1 GiB
```

> [!TIP]
> Read these numbers from the model's `config.json` on Hugging Face: `num_hidden_layers`, `num_key_value_heads`, and `head_dim` (or `hidden_size / num_attention_heads` when `head_dim` is absent). Use the KV head count, not the attention head count: models with grouped-query attention share each KV head across several query heads, which shrinks the cache.

Four concurrent 8K requests need about 4.3 GB of KV cache *on top of* the weights. A small calculator makes the trade-offs concrete:

```python
GB = 1e9

def weights_gb(params_billion: float, bits_per_weight: float) -> float:
    return params_billion * 1e9 * bits_per_weight / 8 / GB

def kv_cache_gb(layers: int, kv_heads: int, head_dim: int, tokens: int, bytes_per_value: int = 2) -> float:
    # 2x for keys and values, per layer, per KV head, per dimension, per token
    return 2 * layers * kv_heads * head_dim * bytes_per_value * tokens / GB

def total_gb(params_billion, bits, layers, kv_heads, head_dim, context, concurrent, overhead=0.2):
    w = weights_gb(params_billion, bits)
    kv = kv_cache_gb(layers, kv_heads, head_dim, context * concurrent)
    return w, kv, (w + kv) * (1 + overhead)

# Llama 3.1 8B: 32 layers, 8 KV heads, head_dim 128
for label, bits in [("BF16", 16), ("8-bit", 8), ("4-bit (~4.8 bpw)", 4.8)]:
    w, kv, total = total_gb(8.0, bits, 32, 8, 128, context=8192, concurrent=4)
    print(f"8B {label:<17} weights {w:5.1f} GB + KV {kv:4.1f} GB -> ~{total:5.1f} GB")
```

```text
8B BF16              weights  16.0 GB + KV  4.3 GB -> ~ 24.4 GB
8B 8-bit             weights   8.0 GB + KV  4.3 GB -> ~ 14.8 GB
8B 4-bit (~4.8 bpw)  weights   4.8 GB + KV  4.3 GB -> ~ 10.9 GB
```

So the 8B model at BF16 just misses a 24 GB card with this workload, while 8-bit fits comfortably and 4-bit fits a 12 GB consumer GPU. Scale up to **Llama 3.1 70B** (80 layers, 8 KV heads, head dimension 128): BF16 weights alone are 140 GB; add the KV cache and the 20% overhead and you need about 181 GB, so plan for four 80 GB GPUs with tensor parallelism (it splits the 64 attention heads evenly, so 2, 4 or 8 GPUs, not 3), while 4-bit weights are about 42 GB and, with four 8K requests (10.7 GB of KV cache), fit on one 80 GB GPU.

The quick rule of thumb: **about 2 GB per billion parameters at 16-bit, 1 GB at 8-bit, 0.6 GB at 4-bit, plus KV cache for your context length times concurrency, plus 20%.** Notice that KV cache grows with context *and* concurrency: long contexts are what usually exhaust memory, not the weights.

## Throughput vs latency

Generation has two phases. **Prefill** processes the whole prompt in parallel and is limited by compute; it determines time to first token. **Decode** produces output one token at a time, and each step must read every weight from GPU memory, so a single request's speed is limited by memory bandwidth:

```text
max tokens/s for one request ~= memory bandwidth / bytes of weights read per token
RTX 4090 (~1,008 GB/s), 8B at 4-bit (~4.8 GB):  ~210 tokens/s upper bound
H100 SXM (~3,350 GB/s), 8B at BF16 (~16 GB):    ~210 tokens/s upper bound
```

Real numbers are lower, but the shape holds: smaller or more quantized weights decode faster. Batching changes the picture. Decoding 32 requests together reads the weights once per step for all of them, so total **throughput** (tokens per second across all users) rises dramatically, while each individual request gets somewhat slower (higher **latency**). The knobs pull in opposite directions:

| Goal | Levers |
| --- | --- |
| Lowest latency per user | Smaller or quantized model, low `--max-num-seqs`, short prompts |
| Highest throughput and lowest cost per token | Large batches (high concurrency), enough KV cache memory, quantization |
| Long contexts | More memory per request: lower concurrency, FP8 KV cache, more GPUs |

Measure with a load test that mimics your traffic: send requests with realistic prompt and output lengths at increasing concurrency, and record time to first token, total latency at p50 and p95, and aggregate tokens per second. Pick the concurrency where p95 latency is still acceptable.

## Running it in production

A model server is a stateful, expensive service. Treat it like one:

- **Security**: require an API key (`--api-key`) or put the server behind your authenticated gateway; never expose it directly to the internet.
- **Health and metrics**: vLLM serves `/health` for load-balancer checks and Prometheus metrics at `/metrics` (running and waiting requests, KV cache usage, token throughput). Alert when requests queue or the cache stays full.
- **Capacity and back-pressure**: when the server is saturated, requests queue. Set client timeouts, cap concurrency in your gateway, and return a clear "busy" error instead of letting requests pile up.
- **Retries and fallback**: retry connection errors and 5xx with backoff from the client; consider falling back to a hosted API (for data that is allowed to leave) when your cluster is down or overloaded.
- **Cold starts**: loading a large model takes minutes. Keep warm replicas and scale before traffic arrives, not after.
- **Model updates**: pin the exact model repository and revision, run your eval on any new version or quantization, and roll out gradually.
- **Licences**: open-weight does not mean unrestricted. Check each model's licence for commercial use and attribution requirements.

## Practice

1. Install Ollama, pull `llama3.1:8b`, and call it through both `/api/chat` and `/v1/chat/completions`. Record tokens per second from `eval_count` and `eval_duration`.
2. Raise `num_ctx` to 16384, send a long prompt, and watch `ollama ps` to see the memory change.
3. Size this deployment by hand: a 14.8B-parameter model with 48 layers, 8 KV heads and head dimension 128, at 4-bit (4.8 bits per weight), serving 4 concurrent requests of 16,384 tokens each with a 16-bit KV cache. Does it fit on a 24 GB GPU? If not, list two changes that make it fit.

**Solution**

Parts 1 and 2 depend on your hardware. On a recent laptop GPU, expect tens of tokens per second for an 8B 4-bit model; on CPU only, a few. `ollama ps` shows memory use rising with `num_ctx`, because the KV cache is allocated for the whole context window.

Part 3:

```text
Weights   = 14.8e9 x 4.8 bits / 8          = 8.9 GB
KV/token  = 2 x 48 x 8 x 128 x 2 bytes      = 196,608 bytes (192 KiB)
KV total  = 196,608 x 16,384 x 4 requests  = 12.9 GB
Total     = (8.9 + 12.9) x 1.2 overhead     = ~26.1 GB  -> does not fit in 24 GB
```

The KV cache, not the weights, is the problem. Changes that make it fit:

- Halve concurrency to 2 requests: KV 6.4 GB, total about (8.9 + 6.4) x 1.2 = 18.4 GB.
- Halve the context to 8,192 tokens: the same 6.4 GB of KV and about 18.4 GB total.
- Store the KV cache in 8-bit (in vLLM, `--kv-cache-dtype fp8`): KV 6.4 GB, total about 18.4 GB, with a small quality cost to verify with your eval.

With vLLM you would express the first two as `--max-num-seqs 2` or `--max-model-len 8192`.

## Key takeaways

- Self-host for data control, offline use, custom models or very high steady volume; at small scale hosted APIs are usually cheaper once idle GPUs and engineering time are counted.
- Ollama is the quickest way to run quantized models locally; vLLM serves many concurrent users efficiently with continuous batching. Both offer an OpenAI-compatible `/v1/chat/completions` endpoint.
- Quantization trades a little quality for a lot of memory: 8-bit is nearly lossless, 4-bit is usually fine, below that degrades; evaluate on your tasks.
- GPU memory = weights (about 2 / 1 / 0.6 GB per billion parameters at 16 / 8 / 4 bits) + KV cache (grows with context length times concurrency) + about 20% overhead.
- Batching raises throughput but adds per-request latency; secure, monitor, load-test and capacity-plan the server like any production service.

## Quiz

1. Roughly how much GPU memory do the weights of a 30-billion-parameter model need at 4-bit quantization?

   A. About 60 GB
   B. About 30 GB
   C. About 18 GB
   D. About 4 GB

   **Answer: C.** 4-bit formats average a little over 4 bits per weight, about 0.6 bytes per parameter: 30 x 0.6 is about 18 GB. 60 GB is the 16-bit figure and 30 GB the 8-bit figure. KV cache and overhead come on top.

2. Your vLLM server runs out of memory when you raise --max-model-len from 8K to 32K, though the weights fit easily. Why?

   A. Longer contexts require reloading the weights
   B. The KV cache grows with context length per request, so each request now reserves about four times as much memory
   C. vLLM stores a copy of the weights per request
   D. Quantization is disabled for long contexts

   **Answer: B.** KV cache size is proportional to tokens times concurrency. Quadrupling context quadruples KV memory per request. Reduce concurrency, use an 8-bit KV cache, or add GPU memory.

3. Increasing the number of concurrent requests batched on one GPU usually does what?

   A. Raises total throughput while making each individual request somewhat slower
   B. Lowers both throughput and latency
   C. Has no effect because decode is compute-bound
   D. Makes each request faster because the GPU is warmer

   **Answer: A.** Decode is memory-bandwidth-bound: each step reads all the weights. Batching shares that read across many requests, multiplying aggregate tokens per second, but each request waits on a bigger step, so per-request latency rises.

4. Which is the strongest reason to self-host instead of using a hosted API?

   A. You expect a few hundred requests per day
   B. You want to avoid writing evals
   C. Open models are always higher quality than hosted ones
   D. Regulations require that the data never leaves your own network

   **Answer: D.** Data residency and offline requirements are classic reasons. Low volume favours hosted per-token pricing, open models are usually behind frontier models on hard tasks, and you still need evals to compare them.