Cache stampedes

A cache stampede (thundering herd, dogpile) happens when a popular key expires or is evicted and many concurrent requests miss at the same moment. All of them recompute the value or query the database for the same row, and the burst can overload the backend the cache was protecting.

#Request coalescing

Let only one request per key load the value; the others wait for its result.

var group singleflight.Group // golang.org/x/sync/singleflight

func (c *Cache) Get(ctx context.Context, key string) (Product, error) {
	if p, ok := c.lookup(ctx, key); ok {
		return p, nil
	}
	v, err, _ := group.Do(key, func() (any, error) {
		p, err := c.db.Load(ctx, key)
		if err == nil {
			c.store(ctx, key, p)
		}
		return p, err
	})
	if err != nil {
		return Product{}, err
	}
	return v.(Product), nil
}

singleflight works within one process; across N servers you get at most N loads. A short-lived distributed lock (SET lock:key 1 NX PX 5000 in Redis) extends it across servers, with waiters retrying the cache.

#Refresh before expiry

• Refresh-ahead: a background job reloads known hot keys before their TTL ends.
• Probabilistic early expiration: each read recomputes early with a probability that rises as expiry approaches, so typically one request refreshes in time and nobody sees a miss.
• Stale-while-revalidate: keep serving the expired value while one request refreshes it in the background. HTTP supports it directly: Cache-Control: max-age=60, stale-while-revalidate=30.

#Spread out expiry

Keys written together with the same TTL expire together. Add random jitter (±10%) to TTLs so a cache warm-up doesn't become a synchronised wave of misses later.

#Also plan for the empty cache

A cache restart is a stampede on every key at once. Make sure the backend survives a cold cache (or warm it first), and use load shedding to cap what reaches the database while the cache refills. See hot partitions for a single key that's hot all the time.