System Design Lab›System Design Questions›Design Distributed Message Queue (Kafka)
Design Distributed Message Queue (Kafka)
HardMessaging Systemsqueuemessagingdistributedstreamingreplication
Question Overview
Design a Kafka-like distributed log where producers append records to partitioned topics and many consumer groups read them independently. Key challenges include replicated append-only storage, per-partition ordering, leader failover without losing acknowledged data, and choosing the right delivery semantics.…
Sign up to see the full question and AI interviewer
Requirements
- Topics split into partitions; records with the same key always go to the same partition
- Ordering guaranteed within a partition, with each record addressed by a monotonically increasing offset
- Consumer groups share partitions so each partition has exactly one active reader per group
- Consumers commit offsets and can replay from any offset still within retention
- No loss of acknowledged records when a broker fails (acks=all with min.insync.replicas = 2)
- Time- or size-based retention and optional per-key log compaction
Back-of-the-envelope numbers
- Ingest: 1M records/s × 1 KB = 1 GB/s average, 3 GB/s at 3× peak
- Replication: RF = 3 adds 2 follower copies → 2 GB/s average (6 GB/s peak) of inter-broker traffic and 3 GB/s of total disk writes
- Consumer egress: 3 groups × 1 GB/s = 3 GB/s average, 9 GB/s at peak, mostly served from the OS page cache
- Storage: 1 GB/s × 86,400 s ≈ 86.4 TB/day × 7 days ≈ 605 TB × 3 replicas ≈ 1.8 PB before compression
- Brokers: 1.8 PB ÷ ~30 TB usable per broker ≈ 60 brokers, so storage rather than throughput sets the cluster size
- Per-broker network at peak: (3 + 6 GB/s) ÷ 60 ≈ 150 MB/s in, (6 + 9 GB/s) ÷ 60 ≈ 250 MB/s out, fine for a 25 Gbps NIC
- Partitions: at ~10 MB/s per partition a 100 MB/s topic needs ≥ 10; 20K partitions × 3 replicas ÷ 60 brokers ≈ 1K replicas per broker
Key components
- Partition log: an append-only sequence of segment files (e.g. 1 GB each) with sparse offset and timestamp indexes, written sequentially
- Broker I/O path: the OS page cache and zero-copy sendfile serve consumers, and durability comes from replication rather than per-message fsync
- Replication: one leader per partition with followers fetching from it; the in-sync replica set (ISR) and high watermark define committed, visible records
- Controller quorum (KRaft, Raft-based): stores topic and partition metadata, tracks broker liveness, and elects a new leader from the ISR on failure
- Producer: key-hash partitioner, batching via linger.ms and batch.size, compression, and an idempotent producer ID plus sequence numbers to drop retried duplicates
- Consumer group coordinator: assigns partitions to members, runs rebalances, and stores committed offsets in an internal compacted topic
- Retention and compaction: expired data is dropped by deleting whole segments, while the compactor rewrites segments keeping only the latest record per key
Common mistakes
- Promising global ordering across a topic, when Kafka-style systems only order records within a single partition
- Using acks=1 while claiming no data loss; the leader can acknowledge and then fail before any follower copies the record
- Having brokers track per-message acknowledgements and deletions like a classic queue instead of letting consumers own their offsets
- Adding more consumers than partitions and expecting more parallelism, when the extra consumers simply sit idle
- Increasing the partition count of a keyed topic without realizing it changes the key-to-partition mapping and breaks per-key ordering
- Claiming end-to-end exactly-once without transactional writes or idempotent sinks; broker-side exactly-once covers only Kafka-to-Kafka pipelines
- Enabling unclean leader election for availability without acknowledging that it can silently drop committed records
Likely follow-ups
- How do you implement exactly-once processing for a consume-transform-produce pipeline?
- What happens if a leader fails and the only surviving replica has fallen out of the ISR?
- A consumer group falls a day behind; how do you detect it, and what happens when retention deletes unread data?
- How would you cut storage cost for 90-day retention without adding more brokers?
- How would you replicate topics to a second region for disaster recovery, and what happens to consumer offsets?
- How would you support delayed delivery or per-message retries with a dead-letter queue?
No community solutions yet
Be the first to publish your solution
Practice ‘Design Distributed Message Queue (Kafka)’ with an AI Interviewer
Get scored feedback on your diagram, scalability approach, and trade-offs. Free while we grow — up to 3 full interviews a day.