Flipkart ran roughly 900 MySQL clusters, and every one of them had the same dirty secret: when the primary node died, users felt it. Their fix wasn't a better failover. It was removing the concept of "the primary" entirely.
At 1M+ queries per second, the old shape — a primary database with standby replicas — meant that every primary failure caused a short outage while a replica was promoted. Multiply that fragility across ~900 clusters and "zero downtime" is impossible by construction.
The classic database setup has one primary that takes writes and replicas that copy it. If the primary dies, the system must notice, pick a replica, and promote it — and during that handoff, writes stall. It's brief, but it's a real outage, and it happens every time hardware fails or you do maintenance.
Distributed SQL databases remove the single primary. Data is spread across peer nodes that agree via consensus, so losing any one node doesn't require a promotion — the remaining nodes simply carry on. Failure stops being an event and becomes a non-event.
Flipkart migrated to TiDB, a distributed SQL database, running on Kubernetes. Because TiDB nodes are peers coordinated by a consensus protocol (Raft) rather than a primary with hot standbys, the cluster tolerates node loss without a write-stalling failover. To operate it at their scale, Flipkart built a custom Flipkart Operator that automates the risky lifecycle work — draining a node, upgrading it, rejoining it, and rebalancing data — and schedules it in the 2–3 AM low-traffic window, one node at a time.
Worth knowing
The Operator is the unsung hero. Distributed SQL gives you the capability for zero-downtime maintenance, but at 900-clusters scale you can't drain and rebalance nodes by hand — you'd never finish, and you'd make mistakes. Encoding that operational knowledge into automation is what converts a theoretical property into a reliable practice. The database choice is half the story; the operator is the other half.
The gap it reveals
Most engineers stop at "add a replica for high availability." The deeper realisation is that primary/replica failover is itself a source of downtime, and that distributed-SQL consensus eliminates the promotion step — but only pays off operationally if you automate the node lifecycle. Connecting the database model to the maintenance-window automation is the systems-at-scale view.
In the interview room
"How do you achieve zero-downtime database maintenance?" trips up candidates who only know primary/replica. The strong answer names the failover stall as the enemy, proposes consensus-based peers (no single primary), and adds rolling, automated, off-peak node replacement. Mentioning that you'd schedule drains in a low-traffic window shows you think operationally, not just architecturally.
The reframe
High availability isn't about recovering from failure quickly — it's about designing failure to be uneventful. As long as your architecture has a "the primary" whose death triggers a scramble, you have a built-in outage waiting for a hardware fault. Flipkart's win was making node loss boring.
The most available systems don't fail over faster. They have nothing to fail over to — because no node was ever in charge.
Primary source →
pingcap.com — How Flipkart Scales Over 1M QPS with Zero Downtime Maintenance