DoorDash cut their cloud bill so suddenly that their cloud provider called to ask if something had broken. Nothing had broken. They'd just stopped paying for a line item nobody draws on architecture diagrams.
When DoorDash moved from a monolith to microservices, two costs crept up together: blast radius (more services, more ways to fail) and a quieter one — cross-availability-zone data transfer. Cloud providers charge per gigabyte every time data crosses between zones, and a microservice mesh sends data across zones constantly. The architecture diagram showed clean boxes and arrows. The invoice showed the arrows had a price.
To survive a data-center problem, clouds split each region into multiple availability zones (separate buildings, essentially), and you run copies of your services in each. Good for reliability. But moving data between zones costs money — a per-gigabyte fee.
In a monolith, internal calls happen inside one process: free. Split into microservices and those calls become network requests — and if service A in zone 1 happens to call service B in zone 2, you're billed for it. Multiply by thousands of services chatting millions of times a second, and you're paying a fortune to shuffle your own data between your own buildings.
The cost that hides in the topology
This is the insidious part: nothing is wrong. Every service is healthy, every request succeeds. The cost comes purely from where the talking partners happen to live. Default routing picks a target service instance more or less at random across all zones, so most calls needlessly hop a zone boundary — each hop metered. DoorDash found that 90% of their traffic was HTTP1/HTTP2/gRPC service-to-service chatter, exactly the traffic that was crossing zones for no reason.
The fix is zone-aware routing via their Envoy-based service mesh. Envoy is configured to prefer sending a request to an instance of the target service in the same zone as the caller, only crossing zones when the local instance is unavailable. The data stays put; the cross-AZ fee evaporates.
For this to be safe you need an instance of every service in every zone — otherwise "prefer local" has nothing local to prefer. DoorDash uses Kubernetes topologySpreadConstraints to guarantee each service's pods are evenly spread across zones, and structures everything as a cell-based architecture: one deployment per cell, with no intercellular traffic. Same reliability posture (still multi-zone), dramatically lower transfer cost.
Worth knowing
That phone call is the whole story in miniature. A cost reduction large enough to look like an outage from the provider's side means the baseline waste was enormous — and invisible, because it never showed up as latency or errors. Cross-AZ transfer is one of the largest hidden line items in cloud spend precisely because it's a correctness-neutral cost: everything works perfectly while you bleed money.
The gap it reveals
System design interviews and textbooks treat the network as free and instant. The senior realisation is that in a real cloud, data movement has a dollar cost that depends on topology — and that a microservices migration can quietly multiply your bill without any visible degradation. Knowing to make routing zone-aware, and why you need topology spread to do it safely, is cost-reasoning most engineers never develop because the meter is invisible.
In the interview room
Cost is the dimension almost no candidate volunteers, which is exactly why it impresses. When you draw a multi-AZ microservices deployment, add: "I'd make the mesh zone-aware so service-to-service calls stay in-zone, since cross-AZ transfer is billed per GB." That one sentence signals you've owned a cloud bill, not just a whiteboard.
The reframe
Architecture diagrams lie by omission: they draw arrows as if data movement were free. It isn't. Every line between two boxes in different zones is a recurring charge, and the most expensive systems are often the ones that work flawlessly while quietly hemorrhaging money on correctness-neutral overhead. DoorDash's win wasn't making anything faster or more reliable — it was learning to see the cost that was always there.
The cheapest optimisation is the one that changes nothing a user can see.
Primary source →
careersatdoordash.com — Staying in the Zone: Using a Service Mesh to Manage Data Transfer