Here is a fact that breaks most people’s intuition: for certain queries, adding more servers makes them slower. LinkedIn ran straight into it, and the culprit is the most underrated number in distributed systems — the tail.
The feature was “People You May Know”-style scoring in Venice, LinkedIn's derived-data store. A single request is a large multi-key lookup that has to touch nearly every partition — and therefore nearly every server — to assemble its answer. As the cluster grew horizontally, p99 latency got worse, not better. The thing they did to handle more load was deepening the latency problem.
A fan-out (or scatter-gather) request splits into many sub-requests sent to many servers, and can't finish until the last one comes back. That “wait for the slowest” rule is the whole story. Your request is only as fast as the unluckiest server it touched.
Now the trap: every server occasionally has a slow moment — a garbage-collection pause, a disk hiccup. The more servers a single request must wait on, the higher the odds that at least one of them is having a bad moment right now. Add servers to handle load, and you add chances to get unlucky. The average stays fine; the tail gets vicious.
The math that should be tattooed on every architect
Say each server responds slowly just 1% of the time. A request touching one server is slow 1% of the time — no problem. But a request that fans out to 100 servers is slow whenever any of them is slow: 1 − (0.99)^100 ≈ 63%. The same per-server reliability that's invisible at small scale becomes the dominant experience at large fan-out. This is tail latency amplification, and it's why scaling out a fan-out workload can backfire.
Because you can't fix this by adding hardware, LinkedIn attacked the tail directly. Partition-assignment optimizations reduce how many servers a single request must touch — fewer participants, smaller chance of an unlucky one. Network-bandwidth reductions cut the per-request cost so each sub-request is lighter. A migration to RocksDB cut garbage-collection pauses, one of the biggest sources of random per-node slowness (the same GC-tail problem Discord hit). And they evaluated HTTP/2 to multiplex sub-requests more efficiently.
Notice the through-line: every fix either reduces the number of nodes touched or reduces each node's probability of a slow moment. Those are the only two levers against fan-out tail. Throughput — the thing horizontal scaling buys — isn't on the list.
The gap it reveals
Engineers reach for “scale horizontally” as a reflex. The realization that separates seniors is that for fan-out workloads, scaling out can amplify tail latency rather than reduce it — because the request waits on the slowest of an ever-growing set. Knowing the probability math (1 − (1−p)^n) and that the fixes target node count and per-node tail, not throughput, is genuinely advanced.
In the interview room
Any scatter-gather design (search, feed scoring, recommendations) invites: “what happens to p99 as you add nodes?” The naive answer assumes it improves. The strong answer: “for fan-out, p99 can degrade with more nodes due to tail amplification, so I'd limit nodes-touched, hedge slow sub-requests, and attack per-node tail sources like GC.” That single insight signals deep distributed-systems maturity.
The reframe
“Just scale horizontally” is the most reflexive advice in our field, and for fan-out workloads it's a trap. Throughput and tail latency are different problems with opposite responses to scaling out. LinkedIn's Venice work is a reminder that the average hides the experience — your users live in the tail, and the tail does not get better just because you added boxes.
For fan-out, the enemy isn't load. It's the slowest node you can't stop touching.
Primary source →
engineering.linkedin.com — Supporting Large Fanout Use Cases at Scale in Venice