Everyone obsesses over which database scales. Almost nobody asks the question that actually broke Discord: what happens when one row gets too popular?
Discord stores trillions of messages. The headline number is impressive, but it's a distraction. The thing that kept their engineers up at night wasn't total volume — it was a single channel, lighting up at a single moment, on a single node.
Imagine a library where every book lives on exactly one shelf, and the librarian for that shelf is the only person allowed to fetch it. Most shelves are quiet. But when a celebrity tweets a link to one specific book, ten thousand people show up wanting that exact title — all routed to one overwhelmed librarian. The rest of the library is empty. Doesn't matter. The line is out the door.
That's a hot partition. Your data is spread evenly across machines, but your traffic isn't. And the machine doesn't care that it's only responsible for 0.001% of your data — it's still pinned at 100% CPU while its neighbours nap.
What actually went wrong
Discord ran Cassandra across 177 nodes. Cassandra (and its JVM) periodically pauses to run garbage collection. Under a hot partition, those pauses stack onto an already-saturated node. And because Discord read at quorum — waiting for a majority of replicas to agree — a single slow replica dragged the latency of the whole read up. Their p99 reads swung between 40 and 125 milliseconds; writes, 5 to 70. Unpredictable tail latency, cluster-wide, triggered by one busy channel.
The migration to ScyllaDB gets the headlines — a C++ rewrite of Cassandra with a shard-per-core architecture and, crucially, no garbage collector. Each CPU core owns a slice of data and runs its own event loop, so there's no stop-the-world pause to amplify. They even built a custom "superdisk": local NVMe SSDs RAIDed with a network persistent disk to get speed without losing durability.
But the migration to a faster engine isn't the lesson. The lesson is what they put in front of it.
Request coalescing — the real fix for hot keys
Discord inserted intermediary data services, written in Rust, between the API and the database. Requests are routed by channel ID using consistent hashing, so all reads for one channel land on the same data-service instance. And there, the trick: if 10,000 users request the same channel at the same instant, the data service fires one query to the database and makes the other 9,999 wait on that single result.
// conceptually, per hot key: if inflight.contains(key): await inflight[key] # piggyback on the in-flight read else: inflight[key] = db.read(key) # exactly one real DB hit result = await inflight[key] inflight.remove(key)
This is the move people miss. A faster database raises the ceiling. Coalescing collapses the load before it ever reaches the database. One of those is buying hardware; the other is understanding your traffic.
Then the migration itself
Moving trillions of rows with a naive single-threaded copy was projected at three months. They rewrote the migrator in Rust to drive massive concurrency and hit 3.2 million records per second — finishing in nine days. The cluster went from 177 Cassandra nodes to 72 ScyllaDB nodes, with p99 reads settling around a steady 15ms.
The gap it reveals
Most engineers can define "hot partition." Far fewer can tell you, cold, why quorum reads turn one hot node into a cluster-wide latency event — or that the durable fix is a coalescing layer, not a bigger box. Knowing the term and being able to design around it are different skills. The second one is what gets you the senior offer.
In the interview room
When you reach for Cassandra/Scylla in a design round, the interviewer's real question is hiding behind your choice: "what's your hottest key, and what happens to it?" Answer with consistent-hash routing plus request coalescing before they ask, and you've just demonstrated production judgment instead of pattern recall.
The reframe
Scale isn't about the size of your data. It's about the distribution of your load. Discord's data was always evenly spread — it was the traffic that clustered. The engineers who scale systems aren't the ones who memorise database benchmarks; they're the ones who, before anything else, ask where the load concentrates and what happens when it does.
A bigger database hides the problem. Understanding your access pattern removes it.
Primary source →
discord.com — How Discord Stores Trillions of Messages