Self-hosting open models

Hosted APIs are the fastest way to ship, but sometimes you need the model on hardware you control: data that may not leave your network, an offline device, a fine-tuned model, or volume high enough that renting GPUs is cheaper than paying per token. Open-weight models (Llama, Qwen, Mistral, Gemma and others) make that possible. This lesson covers when self-hosting is worth it, running models locally with Ollama and serving them with vLLM, how quantization works, how to size GPU memory with real arithmetic, the throughput-versus-latency trade-off, and what it takes to run your own model server in production. (The alternative path in this roadmap covers running hosted APIs in production instead.)

#When self-hosting is worth it

Do the cost arithmetic before you decide. Suppose you rent one 80 GB data-centre GPU for an assumed $2.50 an hour (prices vary widely by provider and commitment). Running all month costs about $1,825. A typical request in your app uses 2,000 input and 300 output tokens:

Hosted small model, $0.10 / $0.50 per million tokens:
  2,000 x 0.10/1M + 300 x 0.50/1M = $0.00035 per request
  $1,825 buys ~5.2 million requests a month (about 2 requests/second, around the clock)

Hosted mid-tier model, $2 / $10 per million tokens:
  2,000 x 2/1M + 300 x 10/1M = $0.007 per request
  $1,825 buys ~260,000 requests a month (about 0.1 requests/second)

So the GPU only saves money if you have steady, high volume and an open model that passes your evaluation as well as the hosted model it replaces, and the comparison ignores the engineering time to run it. Cost alone rarely justifies self-hosting at small scale. Data control, offline operation and custom models more often do.

#Running models locally with Ollama

Ollama is the easiest way to run a model on a laptop or a single server. It downloads quantized models, runs them on a GPU if you have one (or Apple Silicon, or CPU), and exposes an HTTP API.

curl -fsSL https://ollama.com/install.sh | sh   # Linux; on macOS and Windows install the app
ollama pull llama3.1:8b                          # download a model (about 5 GB, 4-bit)
ollama run llama3.1:8b "Explain a B-tree index in two sentences."
ollama list                                      # models on disk
ollama ps                                        # models loaded in memory, and on GPU or CPU

The server listens on http://localhost:11434. Its native chat endpoint:

import requests

response = requests.post(
    "http://localhost:11434/api/chat",
    json={
        "model": "llama3.1:8b",
        "messages": [
            {"role": "system", "content": "You are a concise assistant."},
            {"role": "user", "content": "What is a vector index?"},
        ],
        "stream": False,
        "options": {"num_ctx": 8192},  # context window; the default is small on most GPUs
    },
    timeout=120,
)
response.raise_for_status()
data = response.json()
print(data["message"]["content"])
print(f"{data['eval_count']} tokens at {data['eval_count'] / (data['eval_duration'] / 1e9):.1f} tokens/s")

The default context window depends on VRAM: only 4,096 tokens on GPUs with less than 24 GB (32K and 256K on larger ones); prompts longer than num_ctx are silently truncated, a common cause of "the model ignored my instructions" with RAG prompts. Raising it costs memory, as the sizing section explains.

#Serving with vLLM

Ollama is built for convenience and single users. When many users share a GPU server, use an inference engine designed for throughput. vLLM is the most widely used: it batches concurrent requests continuously (new requests join the running batch at each step instead of waiting for it to finish) and manages the attention cache in small pages (PagedAttention), so far more requests fit in memory at once.

On a Linux machine with an NVIDIA GPU:

pip install vllm
vllm serve Qwen/Qwen2.5-7B-Instruct \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --api-key "$VLLM_API_KEY"

This downloads the weights from Hugging Face (gated models such as Llama require accepting the licence and setting HF_TOKEN) and starts an OpenAI-compatible server on port 8000. Key flags:

• --max-model-len: the longest prompt plus output a request may use. Lower values leave room for more concurrent requests.
• --gpu-memory-utilization: the fraction of GPU memory vLLM claims up front (default 0.9) for weights plus attention cache.
• --tensor-parallel-size 2: split a model too big for one GPU across two.
• --max-num-seqs: cap on concurrently running requests.
• --api-key: require Authorization: Bearer <key> on requests.

Call it with plain HTTP:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $VLLM_API_KEY" \
  -d '{
    "model": "Qwen/Qwen2.5-7B-Instruct",
    "messages": [{"role": "user", "content": "Give me three names for a hiking app."}],
    "max_tokens": 200
  }'

{
  "id": "chatcmpl-...",
  "object": "chat.completion",
  "model": "Qwen/Qwen2.5-7B-Instruct",
  "choices": [
    {
      "index": 0,
      "message": {"role": "assistant", "content": "1. TrailMate\n2. Summit\n3. Pathfinder"},
      "finish_reason": "stop"
    }
  ],
  "usage": {"prompt_tokens": 18, "completion_tokens": 14, "total_tokens": 32}
}

Ollama serves the same format at http://localhost:11434/v1/chat/completions, so one small client function works against both, and against many other servers:

import os
import requests

def chat(base_url: str, model: str, messages: list[dict], max_tokens: int = 500) -> str:
    response = requests.post(
        f"{base_url}/v1/chat/completions",
        headers={"Authorization": f"Bearer {os.environ.get('LLM_API_KEY', 'none')}"},
        json={"model": model, "messages": messages, "max_tokens": max_tokens},
        timeout=60,
    )
    response.raise_for_status()
    choice = response.json()["choices"][0]
    if choice["finish_reason"] == "length":
        raise RuntimeError("output truncated at max_tokens")
    return choice["message"]["content"]

print(chat("http://localhost:11434", "llama3.1:8b", [{"role": "user", "content": "Hi!"}]))

#Quantization

A model's weights are numbers. Stored in 16-bit floating point (FP16 or BF16, the usual release format), each parameter takes 2 bytes, so an 8-billion-parameter model needs 16 GB just for weights. Quantization stores weights with fewer bits, plus small scale factors that map them back to the original range:

Real 4-bit formats average a little over 4 bits per weight once scales are included (Q4_K_M is about 4.8). The formats you will meet:

• GGUF: the file format of llama.cpp, used by Ollama. Runs on CPU, Apple Silicon and GPUs. Names like Q4_K_M (4-bit, medium) and Q8_0 (8-bit) describe the scheme. Ollama's default model tags are typically 4-bit.
• AWQ and GPTQ: 4-bit formats for GPU serving with vLLM. Download a pre-quantized repository, for example vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ, and vLLM detects the format.
• FP8: 8-bit floating point supported natively by recent NVIDIA data-centre GPUs; --quantization fp8 in vLLM.

A useful rule: a larger model at 4-bit usually beats a smaller model at 16-bit for the same memory. But quantization hurts some tasks (precise maths, code, long-context recall) more than others, so run your own eval on the quantized model before deploying it.

#Sizing GPU memory

GPU memory has to hold three things:

1. Weights: parameters x bytes per parameter.
2. KV cache: for every token in every active request, the attention keys and values at every layer. Per token: 2 x layers x kv_heads x head_dim x bytes per value.
3. Overhead: activations, CUDA context, fragmentation. Budget around 20%.

Worked example: Llama 3.1 8B has 32 layers, 8 key-value heads and a head dimension of 128 (from the model's config.json). With a 16-bit KV cache:

KV per token = 2 x 32 layers x 8 heads x 128 dims x 2 bytes = 131,072 bytes = 128 KiB
One 8,192-token request = 8,192 x 128 KiB = 1 GiB

Four concurrent 8K requests need about 4.3 GB of KV cache on top of the weights. A small calculator makes the trade-offs concrete:

GB = 1e9

def weights_gb(params_billion: float, bits_per_weight: float) -> float:
    return params_billion * 1e9 * bits_per_weight / 8 / GB

def kv_cache_gb(layers: int, kv_heads: int, head_dim: int, tokens: int, bytes_per_value: int = 2) -> float:
    # 2x for keys and values, per layer, per KV head, per dimension, per token
    return 2 * layers * kv_heads * head_dim * bytes_per_value * tokens / GB

def total_gb(params_billion, bits, layers, kv_heads, head_dim, context, concurrent, overhead=0.2):
    w = weights_gb(params_billion, bits)
    kv = kv_cache_gb(layers, kv_heads, head_dim, context * concurrent)
    return w, kv, (w + kv) * (1 + overhead)

# Llama 3.1 8B: 32 layers, 8 KV heads, head_dim 128
for label, bits in [("BF16", 16), ("8-bit", 8), ("4-bit (~4.8 bpw)", 4.8)]:
    w, kv, total = total_gb(8.0, bits, 32, 8, 128, context=8192, concurrent=4)
    print(f"8B {label:<17} weights {w:5.1f} GB + KV {kv:4.1f} GB -> ~{total:5.1f} GB")

8B BF16              weights  16.0 GB + KV  4.3 GB -> ~ 24.4 GB
8B 8-bit             weights   8.0 GB + KV  4.3 GB -> ~ 14.8 GB
8B 4-bit (~4.8 bpw)  weights   4.8 GB + KV  4.3 GB -> ~ 10.9 GB

So the 8B model at BF16 just misses a 24 GB card with this workload, while 8-bit fits comfortably and 4-bit fits a 12 GB consumer GPU. Scale up to Llama 3.1 70B (80 layers, 8 KV heads, head dimension 128): BF16 weights alone are 140 GB; add the KV cache and the 20% overhead and you need about 181 GB, so plan for four 80 GB GPUs with tensor parallelism (it splits the 64 attention heads evenly, so 2, 4 or 8 GPUs, not 3), while 4-bit weights are about 42 GB and, with four 8K requests (10.7 GB of KV cache), fit on one 80 GB GPU.

The quick rule of thumb: about 2 GB per billion parameters at 16-bit, 1 GB at 8-bit, 0.6 GB at 4-bit, plus KV cache for your context length times concurrency, plus 20%. Notice that KV cache grows with context and concurrency: long contexts are what usually exhaust memory, not the weights.

#Throughput vs latency

Generation has two phases. Prefill processes the whole prompt in parallel and is limited by compute; it determines time to first token. Decode produces output one token at a time, and each step must read every weight from GPU memory, so a single request's speed is limited by memory bandwidth:

max tokens/s for one request ~= memory bandwidth / bytes of weights read per token
RTX 4090 (~1,008 GB/s), 8B at 4-bit (~4.8 GB):  ~210 tokens/s upper bound
H100 SXM (~3,350 GB/s), 8B at BF16 (~16 GB):    ~210 tokens/s upper bound

Real numbers are lower, but the shape holds: smaller or more quantized weights decode faster. Batching changes the picture. Decoding 32 requests together reads the weights once per step for all of them, so total throughput (tokens per second across all users) rises dramatically, while each individual request gets somewhat slower (higher latency). The knobs pull in opposite directions:

Measure with a load test that mimics your traffic: send requests with realistic prompt and output lengths at increasing concurrency, and record time to first token, total latency at p50 and p95, and aggregate tokens per second. Pick the concurrency where p95 latency is still acceptable.

#Running it in production

A model server is a stateful, expensive service. Treat it like one:

• Security: require an API key (--api-key) or put the server behind your authenticated gateway; never expose it directly to the internet.
• Health and metrics: vLLM serves /health for load-balancer checks and Prometheus metrics at /metrics (running and waiting requests, KV cache usage, token throughput). Alert when requests queue or the cache stays full.
• Capacity and back-pressure: when the server is saturated, requests queue. Set client timeouts, cap concurrency in your gateway, and return a clear "busy" error instead of letting requests pile up.
• Retries and fallback: retry connection errors and 5xx with backoff from the client; consider falling back to a hosted API (for data that is allowed to leave) when your cluster is down or overloaded.
• Cold starts: loading a large model takes minutes. Keep warm replicas and scale before traffic arrives, not after.
• Model updates: pin the exact model repository and revision, run your eval on any new version or quantization, and roll out gradually.
• Licences: open-weight does not mean unrestricted. Check each model's licence for commercial use and attribution requirements.

#Practice

1. Install Ollama, pull llama3.1:8b, and call it through both /api/chat and /v1/chat/completions. Record tokens per second from eval_count and eval_duration.
2. Raise num_ctx to 16384, send a long prompt, and watch ollama ps to see the memory change.
3. Size this deployment by hand: a 14.8B-parameter model with 48 layers, 8 KV heads and head dimension 128, at 4-bit (4.8 bits per weight), serving 4 concurrent requests of 16,384 tokens each with a 16-bit KV cache. Does it fit on a 24 GB GPU? If not, list two changes that make it fit.

Weights   = 14.8e9 x 4.8 bits / 8          = 8.9 GB
KV/token  = 2 x 48 x 8 x 128 x 2 bytes      = 196,608 bytes (192 KiB)
KV total  = 196,608 x 16,384 x 4 requests  = 12.9 GB
Total     = (8.9 + 12.9) x 1.2 overhead     = ~26.1 GB  -> does not fit in 24 GB

#Key takeaways

• Self-host for data control, offline use, custom models or very high steady volume; at small scale hosted APIs are usually cheaper once idle GPUs and engineering time are counted.
• Ollama is the quickest way to run quantized models locally; vLLM serves many concurrent users efficiently with continuous batching. Both offer an OpenAI-compatible /v1/chat/completions endpoint.
• Quantization trades a little quality for a lot of memory: 8-bit is nearly lossless, 4-bit is usually fine, below that degrades; evaluate on your tasks.
• GPU memory = weights (about 2 / 1 / 0.6 GB per billion parameters at 16 / 8 / 4 bits) + KV cache (grows with context length times concurrency) + about 20% overhead.
• Batching raises throughput but adds per-request latency; secure, monitor, load-test and capacity-plan the server like any production service.