We train systems to handle servers that crash. The thing that actually took Slack down was worse: a zone that didn't crash. It just got slow and weird, while every health check insisted it was fine.
On June 30, 2021, Slack hit a single-availability-zone gray failure — intermittent network problems within one AWS zone. Not a clean outage that monitoring catches instantly. A murky, partial degradation that's maddeningly hard for automated failure detectors to call, because the zone keeps answering health checks while quietly corrupting real traffic.
We imagine failure as binary: a machine is up or it's down. Reality is messier. A gray failure is when something is technically alive — responding to pings, passing health checks — but performing terribly: dropping some packets, adding huge latency, succeeding intermittently. It's the colleague who shows up to work but is having a quiet breakdown. From the outside, present. In practice, broken.
This is uniquely dangerous in a microservices system, where a single user action fans out into hundreds of internal calls. If even a handful of those calls route through the sick zone, the whole user request stalls or fails — and your dashboards, watching the zone's health check, show green.
Why you can't just detect it harder
The tempting response is to build smarter detection — better health checks, faster failure detectors. Slack's wiser conclusion was that gray failure is fundamentally hard to detect reliably and quickly, and trying to win that arms race is a losing game. The better move isn't faster detection. It's containment: arrange the system so that a sick zone can only damage a contained slice, and make it trivial for a human to amputate that slice.
Slack migrated critical services to a cellular architecture. Each availability zone becomes a self-contained "cell" holding everything a request needs. The rule that makes it work: a request entering a cell is served entirely within that cell — no cross-zone fan-out. This is the structural opposite of the everything-talks-to-everything mesh that let one gray zone poison requests everywhere.
The operational payoff is a single, blunt, reliable lever: traffic draining. When a cell misbehaves, an operator drains traffic away from it — in 1% increments for safety — and the whole drain completes within about five minutes. Crucially, this is human-triggered. Slack deliberately kept a person in the loop rather than trusting automation to correctly diagnose the very thing (gray failure) that's notoriously hard to diagnose.
Worth knowing
The human-in-the-loop choice is counter-cultural in an industry that worships automation, and it's the most instructive part. Slack reasoned: if automated detection can't reliably tell a gray failure from normal jitter, then automating the response just means automating the wrong call. So they automated the mechanism (fast, safe draining) and left the decision to a human. Automate the lever, not the judgment.
The gap it reveals
Most engineers design for crash failures — the clean kind. Recognising gray failure as a distinct and harder problem, understanding why microservice fan-out amplifies it, and knowing that the answer is blast-radius containment plus a human-pulled lever rather than ever-smarter detection — that's a depth of operational reasoning you only get from having watched a green dashboard lie to you.
In the interview room
When you sketch a multi-AZ deployment, the question that probes seniority is: "what happens if one zone is slow but not down?" If your answer is "health checks remove it," you've walked into the trap. The strong answer names gray failure, explains why detection is unreliable, and proposes cellular isolation with operator-driven draining as the containment strategy.
The reframe
Resilience engineering quietly shifts from preventing failure to limiting it. You will never detect every gray failure in time. So the mature design question isn't "how do we catch this faster?" — it's "when this happens and we don't catch it for a while, how small can we make the damage, and how fast can a human cut it off?"
You can't out-detect ambiguity. You can contain it.
Primary source →
slack.engineering — Slack's Migration to a Cellular Architecture