System Design Lab›System Design Questions›Design Webhook Delivery Service
Design Webhook Delivery Service
EasyMessaging Systemsqueuemessagingnotificationssecurityretries
Question Overview
Design the system that pushes platform events to customer URLs, like Stripe or GitHub webhooks. The interesting parts are guaranteeing at-least-once delivery, retrying with backoff for days, signing payloads, and keeping one broken endpoint from slowing everyone else down.…
Sign up to see the full question and AI interviewer
Requirements
- Customers register HTTPS endpoints and subscribe them to specific event types
- Deliver every matching event at least once to every subscribed endpoint, with a unique event ID for deduplication
- Retry failures with exponential backoff and jitter for up to 3 days, then move them to a dead-letter queue
- Sign each request with an HMAC and timestamp so receivers can verify authenticity and reject replays
- Customers can inspect every delivery attempt and replay events manually
- A slow or failing endpoint never delays other endpoints; first attempt within 5 seconds at p99
Back-of-the-envelope numbers
- Deliveries: 500M/day ÷ 86,400 s ≈ 5.8K requests/s average, ~17K/s at 3× peak
- In-flight requests: ~17K/s × 0.5 s average response time ≈ 8.7K concurrent connections (Little's law), far more if endpoints time out
- Retries: 2% of 500M = 10M failed first attempts/day × ~4 retries ≈ 40M/day ≈ 460 retries/s average
- Payload history: 500M × 2 KB = 1 TB/day ≈ 30 TB for 30 days of replayable events
- Attempt log: ~540M attempts/day × ~300 bytes ≈ 160 GB/day of status codes, latencies, and response snippets
- Backlog from one outage: an endpoint receiving 100 events/s that is down for 6 hours accumulates 100 × 21,600 ≈ 2.2M queued deliveries
- Egress: ~17K/s × 2 KB ≈ 35 MB/s ≈ 280 Mbps at peak, before TLS and header overhead
Key components
- Transactional outbox: the business write and its event row commit together, and a relay publishes events to Kafka so none are lost
- Fan-out service: matches each event to subscribed endpoints and creates one delivery task per (event, endpoint) pair
- Delivery workers: HTTP POST with a ~10 s timeout, an HMAC-SHA256 signature over timestamp and body, and 2xx treated as success
- Retry scheduler: tasks keyed by next_attempt_at with exponential backoff plus jitter, then a dead-letter queue and customer alert
- Per-endpoint isolation: per-endpoint queues or fair scheduling, concurrency caps, and circuit breakers so bad endpoints only slow themselves
- Egress proxies with fixed IPs that block private and link-local address ranges after DNS resolution to prevent SSRF
- Delivery log and dashboard backed by a time-partitioned store, plus a replay API that re-enqueues past events
Common mistakes
- Sending webhooks synchronously inside the business request, so a slow customer endpoint slows down the platform's own API
- Using one shared FIFO queue, where retries for a dead endpoint cause head-of-line blocking for every customer
- Promising exactly-once delivery instead of at-least-once with stable event IDs that receivers use to deduplicate
- Retrying immediately in a tight loop without backoff or jitter, hammering endpoints just as they recover
- Signing only the body without a timestamp, which lets an attacker replay a captured request later
- Not guarding against SSRF, letting customers register URLs that resolve to internal services or cloud metadata endpoints
- Assuming events arrive in order; receivers should use timestamps or refetch current state from the API
Likely follow-ups
- How would you guarantee ordering for events about the same object, and what would it cost?
- How would you let customers rotate signing secrets without rejecting valid deliveries mid-rotation?
- How would you stop one huge customer from starving deliveries for everyone else?
- How would you let a customer replay every event from the last 7 days after an outage on their side?
- What happens if the business transaction commits but publishing the event to the queue fails?
- How would you decide when to automatically disable an endpoint, and how does it get re-enabled?
No community solutions yet
Be the first to publish your solution
Practice ‘Design Webhook Delivery Service’ with an AI Interviewer
Get scored feedback on your diagram, scalability approach, and trade-offs. Free while we grow — up to 3 full interviews a day.