# Caching strategies

> Decide where caches belong, pick a read and write pattern, keep cached data correct, and stop hot keys and stampedes from taking the database down.

System Design roadmap · Data at scale · 19 min read · https://levelcode.org/roadmaps/system-design/caching-strategies

A cache is the cheapest capacity you'll ever buy: a 95% hit rate means the database sees one request in twenty. It is also the easiest way to serve wrong data, and a cache that suddenly empties can take down a database that had quietly come to depend on it. This lesson treats caching as a design decision: where caches go, how reads and writes flow through them, how they stay correct, and how they fail. (The Backend with Go roadmap's Redis lesson shows the cache-aside code in detail; here the focus is the architecture around it.)

## The arithmetic of a cache

What a cache buys you depends on its hit rate *h*:

```text
average latency = h × t_cache + (1 − h) × (t_cache + t_origin)
origin load     = (1 − h) × request rate
```

With a 1 ms cache in front of a 20 ms database:

| Hit rate | Average latency | Origin load |
|---|---|---|
| 50% | 11 ms | 50% of requests |
| 90% | 3 ms | 10% |
| 99% | 1.2 ms | 1% |

Going from 90% to 99% cuts origin load by another 10×. That's why the shape of your traffic matters: caches work best when a small set of items gets most of the reads (product pages, user profiles, configuration), and hardly at all for data read once (a user's own order history from three years ago).

## Where caches live

Requests pass several places where a copy could be kept. Each is faster and closer to the user than the next, but harder to invalidate.

| Layer | Example | Typical lifetime | Invalidation |
|---|---|---|---|
| Browser / client | `Cache-Control: max-age=60` | Seconds to a year | You can't reach it; wait for expiry or change the URL |
| CDN | Cloudflare, CloudFront, Fastly | Seconds to days | Purge API, or versioned URLs |
| Reverse proxy | NGINX, Varnish | Seconds to minutes | Purge, or short TTLs |
| In-process | An LRU map inside each app instance | Seconds | Each instance separately — hard |
| Distributed cache | Redis, Memcached | Minutes to hours | One `DEL` reaches every reader |
| Database | PostgreSQL's buffer cache | Automatic | Always correct |

Two rules follow. **Immutable content belongs at the outer layers** with long lifetimes: put a content hash in the filename (`app.3f9a2c.js`) and cache it for a year, because a new version gets a new URL. **Mutable data belongs in the distributed cache**, where you can invalidate it in one place. See [content delivery networks](https://levelcode.org/concepts/cdn).

## Read and write patterns

There are four standard ways to put a cache in the data path. See [caching patterns](https://levelcode.org/concepts/caching-patterns) for a side-by-side summary.

**Cache-aside (lazy loading).** The application checks the cache, and on a miss reads the database and fills the cache. Writes go to the database, then delete the cache entry. It's the default because the cache holds only what's actually read, and a cache outage degrades performance rather than correctness.

**Read-through.** Like cache-aside, but the cache itself loads from the database on a miss (a library or a caching proxy does it). Application code is simpler; behaviour is the same.

**Write-through.** Every write goes to the cache and the database together, so the cache is always warm for recently written data. The cost is a slower write path and a cache full of items that may never be read.

**Write-behind (write-back).** Writes go to the cache, which acknowledges immediately and flushes to the database later in batches. It absorbs huge write rates — counters, view counts, metrics — but anything not yet flushed is lost if the cache node dies. Only use it for data you can afford to lose or rebuild.

```sequence
title: Cache-aside read, then a write
participants: App, Cache, Database
App -> Cache: GET user:7
Cache --> App: miss
App -> Database: SELECT … WHERE id = 7
Database --> App: row v1
App -> Cache: SET user:7 = v1, TTL 10 min
App -> Database: UPDATE users SET … WHERE id = 7
Database --> App: committed (v2)
App -> Cache: DEL user:7
> Delete rather than overwrite: the next read reloads v2 from the source of truth.
```

## Keeping cached data correct

"There are only two hard things in computer science: cache invalidation and naming things." The joke lasts because the problem is real. Three tools, used together, keep caches honest. See [cache invalidation](https://levelcode.org/concepts/cache-invalidation).

**Every entry gets a TTL.** It's your upper bound on staleness when an invalidation is missed — and invalidations are missed: a process crashes between the database commit and the `DEL`, a network blip drops the delete. Choose the TTL by asking how long the business can tolerate a wrong value.

**Delete on write, after the commit.** Deleting before the commit lets a concurrent reader re-cache the old value from the database before your write lands.

**Know the remaining race.** Even with delete-after-commit, cache-aside has one window:

1. Reader A misses and reads v1 from the database.
2. Writer B updates the row to v2 and deletes the cache key.
3. Reader A, delayed, writes v1 into the cache.

Now the cache holds v1 until the TTL expires. The window is small — A must be slower than the entire write — but at high volume it happens. Fixes, from simple to thorough: keep TTLs short for data where this matters; delete again a second or so after the write (a "delayed double delete"); or use **leases**, where a miss hands out a token and a later delete invalidates it, so A's stale `SET` is rejected. Facebook's memcache paper describes leases for exactly this race.

For keys derived from many rows, such as "the first page of a user's feed", invalidating precisely is hard. **Versioned keys** help: include a version number in the key (`feed:v42:user:7`), bump the version when the underlying data changes, and let old entries expire on their own.

## Eviction: what to drop when memory is full

A cache has fixed memory, so something has to go. See [eviction policies](https://levelcode.org/concepts/cache-eviction).

- **LRU** (least recently used) evicts what hasn't been read for longest. A good default for most workloads.
- **LFU** (least frequently used) keeps items with many hits even if they weren't touched in the last minute. Better when a stable set of items is consistently hot, and resistant to one-off scans pushing everything out.
- **TTL-only** eviction never removes live entries early. That's fine until memory runs out.

In Redis, `maxmemory-policy allkeys-lru` or `allkeys-lfu` is the right setting for a pure cache. The default, `noeviction`, makes writes fail when memory is full — correct for a database, wrong for a cache.

## Hot keys and stampedes

Two failure modes take down databases that "had a cache".

**A stampede** happens when a popular key expires and hundreds of concurrent requests all miss at once, then all query the database for the same row. See [cache stampedes](https://levelcode.org/concepts/cache-stampede). Defences:

- **Request coalescing.** Only one request per key per process loads from the database; the others wait for its result. (Go's `singleflight` does this.)
- **Early refresh.** Refresh the value shortly *before* it expires, so it never actually expires under load. Probabilistic early expiration spreads the refreshes out:

```go
// shouldRefresh reports whether this request should recompute the value early.
// delta is how long recomputing takes; beta > 1 refreshes earlier. As expiry
// nears, the chance grows, so exactly one request usually refreshes in time.
func shouldRefresh(expiry time.Time, delta time.Duration, beta float64) bool {
	// 1 - rand.Float64() is in (0, 1], so the logarithm is always finite.
	early := time.Duration(float64(delta) * beta * -math.Log(1-rand.Float64()))
	return !time.Now().Add(early).Before(expiry)
}
```

- **Serve stale while revalidating.** Return the expired value immediately and refresh it in the background. HTTP has this built in: `Cache-Control: max-age=60, stale-while-revalidate=30`.

**A hot key** is one key read so often that the single cache node holding it saturates — a celebrity's profile, a flash-sale item. Defences: keep a small in-process cache for the hottest keys (milliseconds of staleness are usually fine), or replicate the key under several names (`item:42#1` … `item:42#8`) and pick one at random per read. See [hot partitions](https://levelcode.org/concepts/hot-partitions).

## When the cache fails

Plan for the cache being empty, because one day it will be: a restart, a failover, a deploy that changes the key format. Ask what happens to the database at your peak traffic with a 0% hit rate. If the answer is "it falls over", you need one of:

- enough database capacity to survive the cold start, possibly by shedding load;
- a cache with replicas and persistence, so a failover keeps most of the data;
- warming the cache before sending it traffic;
- [load shedding](https://levelcode.org/concepts/load-shedding) or rate limiting at the edge to cap what reaches the database while the cache refills.

Also give the cache client tight timeouts. A cache that responds in 2 seconds instead of 1 millisecond is worse than one that's down, because every request waits for it.

## Practice

An e-commerce site caches product pages with cache-aside in Redis, TTL 1 hour. Problems reported:

1. After a price change, some customers see the old price for up to an hour.
2. At the start of each flash sale, the database CPU spikes to 100% for about 30 seconds.
3. Redis memory filled up last week, and writes to Redis started failing.

Diagnose each, and propose a fix with its trade-off.

**Solution**

**1. Stale prices for up to an hour.** Either price updates don't invalidate the cache, or an invalidation was missed (crash between commit and delete, or the read/write race), and the 1-hour TTL then decides how long the wrong price survives. Fix: delete the product key after every committed price change, ideally from a reliable mechanism (an outbox or change-data-capture consumer) rather than inline application code; shorten the TTL for price data, for example to 5 minutes, or cache prices separately from the rest of the page with a short TTL. Trade-off: more cache misses and database reads for price lookups.

**2. Database spike when a flash sale starts.** Sale items get a burst of traffic at a known moment. Their keys were either cold (never cached) or expire under load, so many requests miss together — a stampede on hot keys. Fix: warm the sale items into the cache before the sale starts; use request coalescing so each app instance loads a key at most once at a time; refresh hot keys early or serve stale while revalidating; for extremely hot items, add a short in-process cache. Trade-off: some complexity and a few seconds of possible staleness.

**3. Redis full, writes failing.** The instance is using `noeviction` (the default) or a `volatile-*` policy with keys that lack TTLs. Fix: set `maxmemory` explicitly and `maxmemory-policy allkeys-lru` (or `allkeys-lfu` if a stable set of products is consistently popular), make sure every cache write has a TTL, and alert on memory usage and eviction rate. Trade-off: evicted items cost a miss later, but the cache keeps working.

## Key takeaways

- A cache's value is its hit rate: 90% → 99% cuts origin load another 10×.
- Put immutable content at the edge with long lifetimes and versioned URLs; put mutable data in a distributed cache you can invalidate.
- Cache-aside is the default; write-behind trades durability for write throughput.
- Every entry needs a TTL; delete after commit; and know the read/write race that can re-cache a stale value.
- Use LRU or LFU eviction for caches, never `noeviction`.
- Defend against stampedes with coalescing, early refresh or stale-while-revalidate, and plan for the day the cache is empty.

## Quiz

1. A cache sits in front of a database. Hit rate rises from 90% to 99%. What happens to the load reaching the database?

   A. It drops by about 9%
   B. It drops to a tenth of what it was
   C. It stays the same
   D. It doubles because of more cache writes

   **Answer: B.** Origin load is (1 − h) × requests: from 10% of requests to 1%, a 10× reduction. Small changes in hit rate near 100% have large effects.

2. With cache-aside, why delete the cache key after the database commit rather than before?

   A. Deleting before is slower
   B. Redis can't delete keys during a transaction
   C. If you delete first, a concurrent reader can miss, read the old row and re-cache it before your write commits
   D. Deleting after the commit guarantees the cache can never be stale

   **Answer: C.** Deleting first opens a window for the old value to be cached again. Deleting after commit narrows the window but doesn't close it entirely: a slow reader can still write a stale value, which is why TTLs and leases exist.

3. Which pattern is appropriate for counting article views at 50,000 increments per second, where losing a few seconds of counts in a crash is acceptable?

   A. Write-behind
   B. Cache-aside
   C. Read-through
   D. No cache: increment the database row directly

   **Answer: A.** Write-behind absorbs high write rates by acknowledging in the cache and flushing batches later. Its risk — losing unflushed data — is acceptable here. 50,000 row updates per second, concentrated on the few most-read articles, would cause severe lock contention on those rows.

4. A popular key expires and 800 requests miss at once. Which technique specifically prevents this?

   A. Switching from LRU to LFU eviction
   B. Request coalescing or refreshing the value before it expires
   C. Increasing Redis memory
   D. Using write-through caching

   **Answer: B.** This is a stampede. Coalescing makes one request load while others wait; early or probabilistic refresh means the key never expires under load. Eviction policy and memory size don't affect expiry-driven misses.