Pinterest's caching fleet handles 180 million requests a second. It wasn't a traffic spike that knocked it over. It was deploying a routine software update — the most controlled event in all of operations.
The cache is enormous: over 5,000 servers, ~180M requests/sec, ~220 GB/s of throughput across a ~460 TB dataset in roughly 70 clusters. And when they rolled out a new memcached binary — a normal, planned deploy — clients started getting connection-refused errors in spikes. The problem wasn't load. It was the restart itself.
When you update software on a server, you restart the program. For a brief moment, it's not listening. All its clients — here, thousands of routing processes — notice the connection dropped and immediately try to reconnect. At the same instant. That synchronised stampede is a thundering herd.
A freshly restarted server has a small waiting room for incoming connections (the "accept queue"). Thousands of simultaneous reconnections overflow that waiting room, and the server starts refusing connections outright — which the clients read as an error. You caused an outage by doing the safest thing you do all week.
Where the herd actually trips
The specific mechanism: every TCP server has a kernel-level accept queue — connections that have completed the handshake and are waiting to be picked up by the application. Its size is capped by a setting called net.core.somaxconn and the listen backlog. Under a reconnect storm, the queue fills faster than memcached can drain it, the kernel rejects the overflow, and clients see ECONNREFUSED. The default queue size was simply never meant for thousands of simultaneous reconnects.
The first fix is mundane and powerful: raise the accept queue size (net.core.somaxconn and the listen backlog). A larger queue absorbs the reconnect burst rather than refusing it, giving memcached time to work through the backlog. No application change — a kernel parameter that the defaults got wrong for this scale.
The second fix is the one that shows real systems depth. Pinterest moved memcached onto the Linux real-time scheduler (SCHED_FIFO). Normally the OS scheduler can pre-empt memcached's threads to run other work; under load, that means the network thread gets interrupted at exactly the wrong moments, adding latency. Real-time scheduling keeps the critical thread running. The result: client-side p99 latency dropped between 10% and 40%, and because each server now did more useful work per unit of hardware, it shaved close to 10% off the fleet's total cost.
Worth knowing
Notice where every fix lived: the kernel, not the code. The application was fine. The bottleneck was in the operating system's defaults — accept-queue sizing and thread scheduling — which are tuned for a generic server, not for one fielding 180 million requests a second. At extreme scale, the layer you've been ignoring becomes the layer that matters.
The gap it reveals
Most engineers stop at "the cache handles reads." They've never considered that deploying a cache can take it down, that the failure lives in the TCP accept queue, or that the OS scheduler is a tunable performance lever. Knowing that the thundering herd applies to reconnections (not just cache misses), and that the fixes are kernel parameters, is the kind of depth that only comes from operating at a scale where defaults break.
In the interview room
If your design leans on a large cache fleet, a deep interviewer may probe: "how do you deploy a new version without an outage?" Strong signals include staggered/rolling restarts, connection draining, and an awareness that mass reconnection is itself a load event the accept queue must absorb. Mentioning somaxconn unprompted is a flex that lands.
The reframe
We obsess over the steady state — requests per second, cache hit rate — and forget that the transitions are where systems break. A deploy, a restart, a failover: these are moments of synchronised behaviour, and synchronisation is the enemy of stability. Pinterest's lesson is that at scale, you must design the transitions as carefully as the steady state, and that the answers often live one layer below your application.
The traffic didn't break it. The reconnection did. Mind the transitions.
Primary source →
pinterest-engineering — Improving Distributed Caching Performance and Efficiency at Pinterest