Caching strategies

A cache is the cheapest capacity you'll ever buy: a 95% hit rate means the database sees one request in twenty. It is also the easiest way to serve wrong data, and a cache that suddenly empties can take down a database that had quietly come to depend on it. This lesson treats caching as a design decision: where caches go, how reads and writes flow through them, how they stay correct, and how they fail. (The Backend with Go roadmap's Redis lesson shows the cache-aside code in detail; here the focus is the architecture around it.)

#The arithmetic of a cache

What a cache buys you depends on its hit rate h:

average latency = h × t_cache + (1 − h) × (t_cache + t_origin)
origin load     = (1 − h) × request rate

With a 1 ms cache in front of a 20 ms database:

Going from 90% to 99% cuts origin load by another 10×. That's why the shape of your traffic matters: caches work best when a small set of items gets most of the reads (product pages, user profiles, configuration), and hardly at all for data read once (a user's own order history from three years ago).

#Where caches live

Requests pass several places where a copy could be kept. Each is faster and closer to the user than the next, but harder to invalidate.

Two rules follow. Immutable content belongs at the outer layers with long lifetimes: put a content hash in the filename (app.3f9a2c.js) and cache it for a year, because a new version gets a new URL. Mutable data belongs in the distributed cache, where you can invalidate it in one place. See content delivery networks.

#Read and write patterns

There are four standard ways to put a cache in the data path. See caching patterns for a side-by-side summary.

Cache-aside (lazy loading). The application checks the cache, and on a miss reads the database and fills the cache. Writes go to the database, then delete the cache entry. It's the default because the cache holds only what's actually read, and a cache outage degrades performance rather than correctness.

Read-through. Like cache-aside, but the cache itself loads from the database on a miss (a library or a caching proxy does it). Application code is simpler; behaviour is the same.

Write-through. Every write goes to the cache and the database together, so the cache is always warm for recently written data. The cost is a slower write path and a cache full of items that may never be read.

Write-behind (write-back). Writes go to the cache, which acknowledges immediately and flushes to the database later in batches. It absorbs huge write rates — counters, view counts, metrics — but anything not yet flushed is lost if the cache node dies. Only use it for data you can afford to lose or rebuild.

#Keeping cached data correct

"There are only two hard things in computer science: cache invalidation and naming things." The joke lasts because the problem is real. Three tools, used together, keep caches honest. See cache invalidation.

Every entry gets a TTL. It's your upper bound on staleness when an invalidation is missed — and invalidations are missed: a process crashes between the database commit and the DEL, a network blip drops the delete. Choose the TTL by asking how long the business can tolerate a wrong value.

Delete on write, after the commit. Deleting before the commit lets a concurrent reader re-cache the old value from the database before your write lands.

Know the remaining race. Even with delete-after-commit, cache-aside has one window:

1. Reader A misses and reads v1 from the database.
2. Writer B updates the row to v2 and deletes the cache key.
3. Reader A, delayed, writes v1 into the cache.

Now the cache holds v1 until the TTL expires. The window is small — A must be slower than the entire write — but at high volume it happens. Fixes, from simple to thorough: keep TTLs short for data where this matters; delete again a second or so after the write (a "delayed double delete"); or use leases, where a miss hands out a token and a later delete invalidates it, so A's stale SET is rejected. Facebook's memcache paper describes leases for exactly this race.

For keys derived from many rows, such as "the first page of a user's feed", invalidating precisely is hard. Versioned keys help: include a version number in the key (feed:v42:user:7), bump the version when the underlying data changes, and let old entries expire on their own.

#Eviction: what to drop when memory is full

A cache has fixed memory, so something has to go. See eviction policies.

• LRU (least recently used) evicts what hasn't been read for longest. A good default for most workloads.
• LFU (least frequently used) keeps items with many hits even if they weren't touched in the last minute. Better when a stable set of items is consistently hot, and resistant to one-off scans pushing everything out.
• TTL-only eviction never removes live entries early. That's fine until memory runs out.

In Redis, maxmemory-policy allkeys-lru or allkeys-lfu is the right setting for a pure cache. The default, noeviction, makes writes fail when memory is full — correct for a database, wrong for a cache.

#Hot keys and stampedes

Two failure modes take down databases that "had a cache".

A stampede happens when a popular key expires and hundreds of concurrent requests all miss at once, then all query the database for the same row. See cache stampedes. Defences:

• Request coalescing. Only one request per key per process loads from the database; the others wait for its result. (Go's singleflight does this.)
• Early refresh. Refresh the value shortly before it expires, so it never actually expires under load. Probabilistic early expiration spreads the refreshes out:

// shouldRefresh reports whether this request should recompute the value early.
// delta is how long recomputing takes; beta > 1 refreshes earlier. As expiry
// nears, the chance grows, so exactly one request usually refreshes in time.
func shouldRefresh(expiry time.Time, delta time.Duration, beta float64) bool {
	// 1 - rand.Float64() is in (0, 1], so the logarithm is always finite.
	early := time.Duration(float64(delta) * beta * -math.Log(1-rand.Float64()))
	return !time.Now().Add(early).Before(expiry)
}

• Serve stale while revalidating. Return the expired value immediately and refresh it in the background. HTTP has this built in: Cache-Control: max-age=60, stale-while-revalidate=30.

A hot key is one key read so often that the single cache node holding it saturates — a celebrity's profile, a flash-sale item. Defences: keep a small in-process cache for the hottest keys (milliseconds of staleness are usually fine), or replicate the key under several names (item:42#1 … item:42#8) and pick one at random per read. See hot partitions.

#When the cache fails

Plan for the cache being empty, because one day it will be: a restart, a failover, a deploy that changes the key format. Ask what happens to the database at your peak traffic with a 0% hit rate. If the answer is "it falls over", you need one of:

• enough database capacity to survive the cold start, possibly by shedding load;
• a cache with replicas and persistence, so a failover keeps most of the data;
• warming the cache before sending it traffic;
• load shedding or rate limiting at the edge to cap what reaches the database while the cache refills.

Also give the cache client tight timeouts. A cache that responds in 2 seconds instead of 1 millisecond is worse than one that's down, because every request waits for it.

#Practice

An e-commerce site caches product pages with cache-aside in Redis, TTL 1 hour. Problems reported:

1. After a price change, some customers see the old price for up to an hour.
2. At the start of each flash sale, the database CPU spikes to 100% for about 30 seconds.
3. Redis memory filled up last week, and writes to Redis started failing.

Diagnose each, and propose a fix with its trade-off.

#Key takeaways

• A cache's value is its hit rate: 90% → 99% cuts origin load another 10×.
• Put immutable content at the edge with long lifetimes and versioned URLs; put mutable data in a distributed cache you can invalidate.
• Cache-aside is the default; write-behind trades durability for write throughput.
• Every entry needs a TTL; delete after commit; and know the read/write race that can re-cache a stale value.
• Use LRU or LFU eviction for caches, never noeviction.
• Defend against stampedes with coalescing, early refresh or stale-while-revalidate, and plan for the day the cache is empty.