Instagram took down their own backend by doing something that sounds completely safe: turning on a fresh, empty cache. The cache was supposed to reduce load. For a few seconds, it did the opposite.
An empty cache means every request is a miss. Bring up a new cache cluster, and suddenly a hundred concurrent requests for the same key all miss simultaneously — and all stampede the backend at once. The thing you added to protect the backend becomes the thing that floods it.
A cache stores computed results so you don't recompute them. When the answer isn't cached (a "miss"), you compute it and store it. Normally fine. But picture a popular key with no value cached yet, and 100 requests arriving in the same instant: all 100 see a miss, all 100 launch the expensive backend computation, and only then does the first result get cached. You did the work 100 times and hammered the backend — a thundering herd.
The naive fix ("lock the key") serialises everyone and adds its own latency. Instagram's fix is cleverer: cache the promise of a result before the result exists.
Instead of caching the value, Instagram caches a future — specifically a folly::Future backed by a SharedPromise. The first caller to miss creates the future and kicks off the single backend request. Every subsequent caller that arrives while it's in flight finds the future already in the cache and simply awaits the same one. When the backend responds, the promise is fulfilled and every waiter is handed the result at once. A hundred concurrent misses collapse into one backend call.
Worth knowing
This is the same insight as Discord's request coalescing — arrived at independently, in a different language, for a different system. That convergence is the tell that it's a fundamental pattern, not a trick: when concurrent callers want the same not-yet-computed thing, cache the computation, not just its output. Spotting the same idea recurring across teardowns is how you build durable mental models.
The gap it reveals
Engineers learn "caches reduce load" and stop there. The gap is realising that a cold cache is a load-amplifier, that cache misses on hot keys cause thundering herds, and that the elegant fix is caching the in-flight promise so concurrent misses dedupe to one backend call. Knowing this turns 'we'll add a cache' from a hand-wave into a real design.
In the interview room
Add a cache in a design round and a sharp interviewer asks: "what happens on a cache miss for a hot key — or when the cache is cold?" The senior answer names the thundering herd and proposes request coalescing / promise-caching (or a single-flight mechanism) so simultaneous misses share one backend call. It shows you've thought past the happy path.
The reframe
Every safeguard has a failure mode that looks exactly like the thing it was meant to prevent. A cache prevents load — until it's empty, when it causes it. Mature engineers don't just add protective layers; they ask how each layer behaves at its worst moment, like the instant it's first switched on. The cold-start is where caches bite.
Don't cache the answer. Cache the asking — so a crowd asks only once.
Primary source →
Instagram Engineering — Thundering Herds & Promises