# Scaling a service

> Grow a service from one machine to many: separate tiers, make servers stateless, add load balancing, read replicas, caches and a CDN — each for a measured reason.

System Design roadmap · Foundations · 18 min read · https://levelcode.org/roadmaps/system-design/scaling-a-service

Almost every large system started as one server running everything. Scaling is the sequence of changes that takes it from there to many machines, and the order of those changes matters: each one removes the current bottleneck and exposes the next. This lesson walks that path step by step, explains what each step costs, and shows how to tell which step you need next.

## Stage 0: one box

A single machine runs the web server, the application and the database. It's cheap, simple to deploy, and trivially consistent: there's one copy of everything. Don't underestimate it. A modern server with 32 cores, 128 GB of RAM and NVMe disks handles thousands of requests per second for many applications.

It has two problems that no amount of tuning fixes:

- **It's a single point of failure.** A kernel update, a full disk or a hardware fault takes everything down.
- **The application and database fight over the same resources.** A CPU-heavy request slows down queries, and a big query starves the web server.

## Stage 1: separate the database

Move the database to its own machine. Now each tier can be sized for its job — the database wants memory and fast disks; the application wants CPU — and you can scale them independently. Most managed database offerings (RDS, Cloud SQL, Azure Database) start you here, with automated backups and a standby replica you can fail over to.

The cost is a network hop on every query: around half a millisecond inside a data centre. That's why patterns like N+1 queries, invisible on one box, suddenly cost 50 ms on a page with 100 queries.

## Stage 2: more application servers

When the application tier runs out of CPU, you have two options (see [vertical vs horizontal scaling](https://levelcode.org/concepts/vertical-vs-horizontal-scaling)):

- **Scale up**: a bigger machine. No code changes, but there's a ceiling, prices rise faster than capacity at the top end, and you still have one machine to fail.
- **Scale out**: more machines of the same size behind a load balancer. No ceiling and better fault tolerance, but it requires the application to be **stateless**.

### Making the application stateless

A server is stateless when any instance can handle any request, because nothing that matters between requests lives in its memory or on its disk. Look for these and move them out:

| State in the server | Where it goes |
|---|---|
| Login sessions in memory | Signed cookies or tokens, or a shared session store such as Redis |
| Uploaded files on local disk | Object storage |
| In-memory caches others depend on | A shared cache, or accept per-instance caches that may disagree |
| Scheduled jobs that must run once | A scheduler with a lock or a dedicated worker |
| WebSocket connections | Allowed, but clients must reconnect to any instance after a failure |

With that done, add a [load balancer](https://levelcode.org/concepts/api-gateway-vs-load-balancer) in front. It spreads requests across instances, health-checks them, and stops sending traffic to failed ones. The DevSecOps roadmap's load-balancing lesson covers the algorithms and health checks in depth; for design purposes the important point is that the load balancer turns "a server died" from an outage into a capacity dip.

> [!TIP]
> Avoid sticky sessions — routing each user to the same instance — as a way to keep state in memory. They make load uneven, and when that instance dies its users lose their sessions anyway. Fix the state instead. See [stateless services](https://levelcode.org/concepts/stateless-services).

## Stage 3: take reads off the primary database

Now the database is usually the bottleneck, and in most applications it's reads that saturate it. Three tools, in the order you usually reach for them:

**Fix the queries first.** Missing indexes, N+1 query patterns and `SELECT *` on wide rows are the most common causes of database load. An index can turn a 2-second scan into a 2-millisecond lookup — a 1,000× improvement no hardware can match.

**Read replicas.** The primary streams every change to one or more replicas, which serve read-only queries. This scales reads roughly linearly, but replicas lag behind the primary by milliseconds to seconds. A user who updates their profile and immediately reloads it may see the old version if the read goes to a replica. The next lessons cover that problem in detail ([replication lag](https://levelcode.org/concepts/replication-lag)); the usual fix is to send a user's reads to the primary for a few seconds after they write.

**A cache.** For data read far more often than it changes, a cache such as Redis in front of the database serves repeated reads in about a millisecond. A 90% hit rate removes 90% of those queries. The caching lessons cover how to keep a cache correct.

```sequence
title: A read in a scaled-out service
participants: Client, Load balancer, App, Cache, Replica
Client -> Load balancer: GET /products/42
Load balancer -> App: forward to a healthy instance
App -> Cache: GET product:42
Cache --> App: miss
App -> Replica: SELECT … WHERE id = 42
Replica --> App: row (possibly a little stale)
App -> Cache: SET product:42, TTL 5 min
App --> Client: 200 product
```

## Stage 4: move static content to the edge

Images, scripts, stylesheets and videos don't need your servers at all. Put them in object storage and serve them through a [CDN](https://levelcode.org/concepts/cdn): a network of caches in hundreds of locations near users. A user in Jakarta then fetches your logo from a server in Jakarta in 10 ms instead of from Virginia in 250 ms, and your servers never see the request.

CDNs can also cache API responses that are the same for everyone (a product catalogue, a public profile) for seconds or minutes. Even a 10-second cache on a hot endpoint cuts origin traffic dramatically during a spike.

## Stage 5: move slow work out of the request

Some work doesn't need to finish before you respond: sending email, resizing images, generating reports, calling slow third-party APIs. Doing it inline makes every request as slow as its slowest step, and ties up a server while it waits.

Put that work on a **queue** and let a separate pool of **workers** process it. The request returns as soon as the job is enqueued, and the worker pool scales independently with queue depth. If a worker crashes, the job is redelivered. The queues lesson covers the guarantees and pitfalls; at this stage, the design point is that a queue also **absorbs bursts**: 10,000 uploads in a minute become a backlog that drains over five minutes instead of 10,000 simultaneous resizes.

## Stage 6: when one database isn't enough for writes

Replicas and caches scale reads. Writes all still go to one primary. When the write rate or the dataset outgrows the largest machine you can buy, you split the data across several databases — **partitioning**, or sharding — so each holds a slice and takes a slice of the writes. This is a big step with real costs: cross-shard queries, rebalancing and harder transactions. Delay it as long as you can, and design for it early by choosing a key that will partition well. The partitioning lesson covers how.

You may also split the *application* into services owned by different teams. That's mostly an organisational decision about deploying independently, not a performance one; see [monolith vs microservices](https://levelcode.org/concepts/monolith-vs-microservices).

## How to know what to do next

Each stage is a response to a measured bottleneck, not a step to take on a schedule. Before scaling, find out which resource is saturated:

| Symptom | Likely bottleneck | Next step |
|---|---|---|
| App CPU near 100%, database idle | Application tier | More instances (if stateless) |
| Database CPU or I/O high, mostly reads | Read load | Fix queries, then replicas or a cache |
| Database write latency rising, lock waits | Write load or contention | Batch writes, fix hot rows, then partition |
| Slow pages, but every server is idle | Latency, not capacity | Find the slow call: N+1 queries, a slow dependency |
| Spikes cause errors, average load is fine | Burst handling | Queues, autoscaling, rate limits |

The [availability](https://levelcode.org/concepts/availability-and-nines) picture improves along the way too. A single box fails as often as its weakest component. With redundant app servers, a database with a standby, and health-checked load balancing, any single machine can fail without taking the service down.

## Practice

A recipe website runs on one server: Rails, PostgreSQL and uploaded photos on the local disk. Traffic is 300 requests per second at peak, 95% reads. Pages take 1.2 seconds at peak; the database CPU sits at 90% while the app CPU is at 40%. A TV appearance next month is expected to bring 10× traffic for a few hours.

Propose the changes, in order, with the reason for each and what you'd measure to confirm it worked.

**Solution**

1. **Find and fix the expensive queries.** Database CPU at 90% with only 300 req/s points at inefficient queries. Use `pg_stat_statements` to find the top queries by total time, add missing indexes, and remove N+1 patterns. Measure: database CPU and p95 page time. This alone may bring page time well under 200 ms.
2. **Separate the database** onto its own machine (or a managed instance with a standby). The app and database stop competing, and you gain a failover target. Measure: database CPU headroom, replication lag on the standby.
3. **Move photos to object storage** and serve them through a CDN. They're probably most of the bytes and many of the requests. Measure: origin request rate, bandwidth.
4. **Make the app stateless** (sessions in a cookie or Redis, no local files) and run at least three instances behind a load balancer. Now an instance failure isn't an outage, and you can add instances for the TV spike. Measure: per-instance CPU, error rate during a deliberate instance kill.
5. **Cache hot pages.** Recipes rarely change: cache rendered recipe pages or API responses in Redis, and give public pages a short CDN TTL (say 60 seconds). During the spike, most requests then never reach Rails. Measure: cache hit rate (aim for over 90%), database queries per second.
6. **Rehearse the spike** with a load test at 10× before the broadcast, and pre-scale app instances rather than relying on autoscaling to react in time.

Not needed: sharding (writes are 5% of 300 req/s), microservices, or a message queue for the spike.

## Key takeaways

- Scale in response to a measured bottleneck; each stage removes one and exposes the next.
- Separate the database, then scale the stateless app tier horizontally behind a load balancer.
- Make servers stateless by moving sessions, files and scheduled jobs out of process memory and local disk.
- Fix queries first, then add read replicas and caches for read load — and handle replica lag explicitly.
- Serve static and cacheable content from a CDN, and move slow, deferrable work onto queues.
- Partition the database only when writes or data size outgrow one machine; it's the most expensive step.

## Quiz

1. Why must application servers be stateless before you can scale them horizontally?

   A. Load balancers can only route to stateless servers
   B. So any instance can serve any request; state kept in one instance's memory or disk is lost or invisible when requests land elsewhere
   C. Stateless servers use less memory
   D. Stateful servers can't connect to a database

   **Answer: B.** With several instances, consecutive requests from one user hit different machines. Sessions, files or jobs kept locally break that, and disappear when the instance dies.

2. A database's CPU is pegged but the application tier is mostly idle. What should you try first?

   A. Add more application servers
   B. Shard the database across four machines
   C. Find the most expensive queries and fix them with indexes or by removing N+1 patterns
   D. Move to microservices

   **Answer: C.** Inefficient queries are the most common cause and the cheapest to fix, often by orders of magnitude. More app servers would add load, and sharding is costly and premature.

3. A user edits their profile and the next page load shows the old name. Reads go to replicas. What is happening?

   A. The write was lost
   B. The read hit a replica that hadn't yet applied the write — replication lag
   C. The CDN is caching the profile
   D. The load balancer is using round robin

   **Answer: B.** Asynchronous replicas trail the primary. A common fix is to route that user's reads to the primary for a short time after they write (read-your-writes).

4. What does putting image resizing on a queue with workers buy, beyond faster responses?

   A. Images are resized at higher quality
   B. Bursts become a backlog that drains at the workers' pace, and the worker pool scales independently of the web tier
   C. Resizing no longer uses CPU
   D. The queue guarantees each image is resized exactly once with no extra work

   **Answer: B.** A queue decouples arrival rate from processing rate, so spikes don't overload anything. Delivery is usually at-least-once, so workers still need to handle duplicates.