Scaling a service

Almost every large system started as one server running everything. Scaling is the sequence of changes that takes it from there to many machines, and the order of those changes matters: each one removes the current bottleneck and exposes the next. This lesson walks that path step by step, explains what each step costs, and shows how to tell which step you need next.

#Stage 0: one box

A single machine runs the web server, the application and the database. It's cheap, simple to deploy, and trivially consistent: there's one copy of everything. Don't underestimate it. A modern server with 32 cores, 128 GB of RAM and NVMe disks handles thousands of requests per second for many applications.

It has two problems that no amount of tuning fixes:

• It's a single point of failure. A kernel update, a full disk or a hardware fault takes everything down.
• The application and database fight over the same resources. A CPU-heavy request slows down queries, and a big query starves the web server.

#Stage 1: separate the database

Move the database to its own machine. Now each tier can be sized for its job — the database wants memory and fast disks; the application wants CPU — and you can scale them independently. Most managed database offerings (RDS, Cloud SQL, Azure Database) start you here, with automated backups and a standby replica you can fail over to.

The cost is a network hop on every query: around half a millisecond inside a data centre. That's why patterns like N+1 queries, invisible on one box, suddenly cost 50 ms on a page with 100 queries.

#Stage 2: more application servers

When the application tier runs out of CPU, you have two options (see vertical vs horizontal scaling):

• Scale up: a bigger machine. No code changes, but there's a ceiling, prices rise faster than capacity at the top end, and you still have one machine to fail.
• Scale out: more machines of the same size behind a load balancer. No ceiling and better fault tolerance, but it requires the application to be stateless.

#Making the application stateless

A server is stateless when any instance can handle any request, because nothing that matters between requests lives in its memory or on its disk. Look for these and move them out:

With that done, add a load balancer in front. It spreads requests across instances, health-checks them, and stops sending traffic to failed ones. The DevSecOps roadmap's load-balancing lesson covers the algorithms and health checks in depth; for design purposes the important point is that the load balancer turns "a server died" from an outage into a capacity dip.

#Stage 3: take reads off the primary database

Now the database is usually the bottleneck, and in most applications it's reads that saturate it. Three tools, in the order you usually reach for them:

Fix the queries first. Missing indexes, N+1 query patterns and SELECT * on wide rows are the most common causes of database load. An index can turn a 2-second scan into a 2-millisecond lookup — a 1,000× improvement no hardware can match.

Read replicas. The primary streams every change to one or more replicas, which serve read-only queries. This scales reads roughly linearly, but replicas lag behind the primary by milliseconds to seconds. A user who updates their profile and immediately reloads it may see the old version if the read goes to a replica. The next lessons cover that problem in detail (replication lag); the usual fix is to send a user's reads to the primary for a few seconds after they write.

A cache. For data read far more often than it changes, a cache such as Redis in front of the database serves repeated reads in about a millisecond. A 90% hit rate removes 90% of those queries. The caching lessons cover how to keep a cache correct.

#Stage 4: move static content to the edge

Images, scripts, stylesheets and videos don't need your servers at all. Put them in object storage and serve them through a CDN: a network of caches in hundreds of locations near users. A user in Jakarta then fetches your logo from a server in Jakarta in 10 ms instead of from Virginia in 250 ms, and your servers never see the request.

CDNs can also cache API responses that are the same for everyone (a product catalogue, a public profile) for seconds or minutes. Even a 10-second cache on a hot endpoint cuts origin traffic dramatically during a spike.

#Stage 5: move slow work out of the request

Some work doesn't need to finish before you respond: sending email, resizing images, generating reports, calling slow third-party APIs. Doing it inline makes every request as slow as its slowest step, and ties up a server while it waits.

Put that work on a queue and let a separate pool of workers process it. The request returns as soon as the job is enqueued, and the worker pool scales independently with queue depth. If a worker crashes, the job is redelivered. The queues lesson covers the guarantees and pitfalls; at this stage, the design point is that a queue also absorbs bursts: 10,000 uploads in a minute become a backlog that drains over five minutes instead of 10,000 simultaneous resizes.

#Stage 6: when one database isn't enough for writes

Replicas and caches scale reads. Writes all still go to one primary. When the write rate or the dataset outgrows the largest machine you can buy, you split the data across several databases — partitioning, or sharding — so each holds a slice and takes a slice of the writes. This is a big step with real costs: cross-shard queries, rebalancing and harder transactions. Delay it as long as you can, and design for it early by choosing a key that will partition well. The partitioning lesson covers how.

You may also split the application into services owned by different teams. That's mostly an organisational decision about deploying independently, not a performance one; see monolith vs microservices.

#How to know what to do next

Each stage is a response to a measured bottleneck, not a step to take on a schedule. Before scaling, find out which resource is saturated:

The availability picture improves along the way too. A single box fails as often as its weakest component. With redundant app servers, a database with a standby, and health-checked load balancing, any single machine can fail without taking the service down.

#Practice

A recipe website runs on one server: Rails, PostgreSQL and uploaded photos on the local disk. Traffic is 300 requests per second at peak, 95% reads. Pages take 1.2 seconds at peak; the database CPU sits at 90% while the app CPU is at 40%. A TV appearance next month is expected to bring 10× traffic for a few hours.

Propose the changes, in order, with the reason for each and what you'd measure to confirm it worked.

Show solution

1. Find and fix the expensive queries. Database CPU at 90% with only 300 req/s points at inefficient queries. Use pg_stat_statements to find the top queries by total time, add missing indexes, and remove N+1 patterns. Measure: database CPU and p95 page time. This alone may bring page time well under 200 ms.
2. Separate the database onto its own machine (or a managed instance with a standby). The app and database stop competing, and you gain a failover target. Measure: database CPU headroom, replication lag on the standby.
3. Move photos to object storage and serve them through a CDN. They're probably most of the bytes and many of the requests. Measure: origin request rate, bandwidth.
4. Make the app stateless (sessions in a cookie or Redis, no local files) and run at least three instances behind a load balancer. Now an instance failure isn't an outage, and you can add instances for the TV spike. Measure: per-instance CPU, error rate during a deliberate instance kill.
5. Cache hot pages. Recipes rarely change: cache rendered recipe pages or API responses in Redis, and give public pages a short CDN TTL (say 60 seconds). During the spike, most requests then never reach Rails. Measure: cache hit rate (aim for over 90%), database queries per second.
6. Rehearse the spike with a load test at 10× before the broadcast, and pre-scale app instances rather than relying on autoscaling to react in time.

Not needed: sharding (writes are 5% of 300 req/s), microservices, or a message queue for the spike.

#Key takeaways

• Scale in response to a measured bottleneck; each stage removes one and exposes the next.
• Separate the database, then scale the stateless app tier horizontally behind a load balancer.
• Make servers stateless by moving sessions, files and scheduled jobs out of process memory and local disk.
• Fix queries first, then add read replicas and caches for read load — and handle replica lag explicitly.
• Serve static and cacheable content from a CDN, and move slow, deferrable work onto queues.
• Partition the database only when writes or data size outgrow one machine; it's the most expensive step.