LinkedIn ran a load test on their own data center and watched their feed's quality-of-service metrics fall off a cliff. The most unsettling discovery wasn't the cliff. It was that the metrics measuring it had been lying by 300%.
In February 2018, a data-center load test made FollowFeed's QoS nosedive. Investigation surfaced three separate problems — and the third one undermined trust in everything: there was no server-side rate limiting, the garbage-collection config was stale, and the error metrics themselves were badly miscounting.
When a server gets more requests than it can handle, it has two options. It can accept everything and try — queues grow, latency explodes, and eventually it collapses, failing all requests. Or it can push back: detect it's over capacity and cleanly reject the excess with a 'try again later' (HTTP 429), so it keeps successfully serving everything it can handle.
That pushing-back is backpressure. A system without it doesn't degrade gracefully — it degrades catastrophically, because it keeps saying yes until it dies.
LinkedIn added server-side request rate limiting so FollowFeed returns 429s when overloaded instead of accepting work it can't complete — converting a potential collapse into clean, bounded shedding. They also re-tuned a stale GC configuration (settings that had quietly fallen out of date as the workload evolved — the same GC-as-hidden-cost theme from Discord's Rust migration).
The most instructive fix was the third: the QoS metrics were overcounting errors by 300%. The dashboards everyone trusted to judge feed health were wrong — which means every prior decision based on them was made on bad data. Rebuilding the metrics wasn't cleanup; it was restoring the team's ability to see reality at all.
Worth knowing
The metrics-were-lying finding deserves a moment. You cannot operate what you cannot measure correctly, and broken observability is more dangerous than broken servers because it's invisible — you make confident decisions on false signals. The highest-value outcome of this load test wasn't a faster feed; it was discovering that the instruments were miscalibrated before an emergency forced LinkedIn to rely on them.
The gap it reveals
Engineers focus on making servers faster; fewer internalise that a server with no backpressure fails all traffic instead of shedding some, and fewer still treat their own metrics as suspect. The realisation that 'return 429 to protect the rest' beats 'accept everything and collapse' — and that you must verify your observability is honest — is operational maturity.
In the interview room
"What happens at peak load?" is an invitation to discuss backpressure. The strong answer: rate-limit server-side and return 429 so you serve within capacity rather than collapsing, and — a rare, impressive addition — 'I'd also confirm my health metrics are accurate, since overload decisions are only as good as the signals behind them.' That second clause is pure senior.
The reframe
Two truths sit inside this teardown. First, a system that can't say no will eventually fail at everything; the ability to shed load cleanly is a feature, not an admission of weakness. Second, your view of your system is only as trustworthy as your instruments — and instruments drift. Test both: that the system pushes back, and that your dashboards tell the truth.
A server that can't say 429 will eventually say 500 to everyone. And a metric you don't audit is a lie you trust.
Primary source →
linkedin.com — Making LinkedIn's Organic Feed Handle Peak Traffic