Every service has a setting for "max concurrent requests." Almost everyone sets it once, to a number someone guessed in 2019, and never touches it again. That stale guess is how systems fall over.
Netflix had a recurring failure shape: a downstream dependency slows down, requests start queuing, queues overflow, timeouts fire, clients retry, and the retries deepen the queue. A cascading failure — born not from too much traffic, but from a concurrency limit that was right yesterday and wrong today.
A concurrency limit is a cap on how many requests a service handles at once. Set it too low and you reject traffic you could have served. Set it too high and, when something slows down, requests pile into a queue faster than they drain — and a queue that only grows is a system that's about to die.
The trouble is there's no single correct number. The right limit depends on how fast requests currently complete, and that changes minute to minute as dependencies speed up and slow down. A fixed setting is a photograph of a moving target.
Why the obvious fix doesn't work
You might think: just load-test the service, find the limit, set it. But the limit isn't a property of the service alone — it's a property of the service and everything it depends on right now. When a database gets slow, the same number of in-flight requests now represents far more queued work. The photograph is already out of date.
Netflix's insight: this is a solved problem, and the solution is sitting in every network connection you've ever made. TCP congestion control doesn't know the network's bandwidth in advance — it infers it. It sends progressively more data until it detects congestion (rising latency, loss), then backs off, continuously probing for the current ceiling.
Their adaptive concurrency limits apply the same idea to service calls. The algorithm watches request latency as a signal: when latency starts climbing, it's a sign the system is approaching its limit, so the cap is reduced. When latency stays low, there's headroom, so the cap is allowed to grow. The limit becomes a living value that tracks reality instead of a constant that decays.
# the gradient idea, simplified gradient = min_observed_rtt / current_rtt # 1.0 = healthy, <1 = slowing new_limit = current_limit * gradient + queue_headroom # latency up -> gradient down -> limit shrinks -> queue drains # latency low -> gradient ~1 -> limit grows -> capacity used
The theory underneath is Little's Law: the average number of in-flight requests equals arrival rate times latency (L = λW). Because W moves, the safe L moves with it. Adaptive limits are just Little's Law, enforced continuously. Netflix open-sourced the implementation as Netflix/concurrency-limits.
The gap it reveals
Most engineers know what a concurrency limit is. The realisation that bites in production is that a static limit is guaranteed to be wrong some of the time, because the correct value is a function of live latency, not a constant. Connecting that to Little's Law — and to TCP congestion control as the proven analogue — is the kind of cross-domain reasoning that marks a senior engineer.
In the interview room
When you draw a service calling a dependency, a sharp interviewer asks: "what happens when the dependency gets slow?" The weak answer is "we set a timeout." The strong answer adds load-shedding via an adaptive concurrency limit, and explains why static limits fail — because the safe limit tracks latency. Reference Little's Law and you've signalled depth most candidates never reach.
The reframe
The deepest fixes in distributed systems are usually borrowed, not invented. Netflix didn't design a novel algorithm — they recognised that protecting a service from overload is the same shape of problem as protecting a network from congestion, and that problem was solved decades ago. The skill wasn't cleverness. It was seeing the pattern across domains.
A constant can't protect you from a variable. Stop guessing the limit; measure it, forever.
Primary source →
netflixtechblog — Performance Under Load: Adaptive Concurrency Limits