System Design Lab›System Design Questions›Design Uptime Monitoring Service
Design Uptime Monitoring Service
EasyInfrastructureschedulingmonitoringnotificationsdistributedtime-series
Question Overview
Design a Pingdom-style service that probes a million websites from around the world and alerts owners when they go down. The interesting parts are scheduling checks evenly without missing any, confirming failures from multiple regions to avoid false alarms, and storing a firehose of results.…
Sign up to see the full question and AI interviewer
Requirements
- Create HTTP(S) checks with URL, interval of 30 s to 5 min, timeout, and expected status or keyword
- Run each check from multiple regions and confirm failures across regions before alerting
- Alert once on down and once on recovery via email, SMS, Slack, or webhook
- Detect a real outage within 2 minutes and never skip a check when a node dies
- Show response-time graphs, uptime percentages, and optional public status pages
- Retain raw results for 30 days and hourly rollups for 2 years
Back-of-the-envelope numbers
- Check rate: 1M checks ÷ 60 s ≈ 16.7K checks/s, steady around the clock if start offsets are spread across each interval
- In-flight probes: 16.7K/s × 0.5 s ≈ 8.3K concurrent requests (Little's law); 1% timing out at 30 s adds ~167 × 30 ≈ 5K more
- Confirmations: with ~1% of checks failing at any moment, 167 failures/s × 2 confirming regions ≈ 330 extra requests/s
- Raw results: 16.7K/s × 86,400 s ≈ 1.44B results/day × 100 B ≈ 144 GB/day ≈ 4.3 TB for 30 days
- Rollups: 1M checks × 8,760 hours/year × ~50 B ≈ 438 GB/year, about 880 GB for 2 years
- Scheduler state: 1M checks × ~200 bytes of config ≈ 200 MB, small enough to hold entirely in memory across scheduler shards
Key components
- Check config store (e.g. PostgreSQL) and API for creating checks, alert contacts, and status pages
- Sharded scheduler: checks hashed into partitions owned via leases; each shard runs a timing wheel keyed by next run with a fixed per-check offset
- Regional probe workers using async HTTP clients that pull jobs from regional queues and record DNS, connect, TLS, and first-byte timings
- Decision engine: per-check state machine (up → suspect → down) that declares down only when at least 2 of 3 regions fail
- Alerting service that deduplicates by state transition, applies escalation and rate limits, and sends via independent providers
- Time-series store (e.g. ClickHouse) partitioned by day with TTLs, plus rollup jobs that produce hourly aggregates
- Meta-monitoring: if one region suddenly fails most of its checks, treat it as a probe problem and suppress alerts
Common mistakes
- Alerting on a single failed request from a single region, which pages customers for transient network blips
- Starting every 1-minute check at the top of the minute, creating a thundering herd on probes and targets
- Polling SELECT ... WHERE next_run <= now() on one database every second, a hot table and single point of failure
- Using a thread per request, so 30-second timeouts during a large outage exhaust the probe's thread pool
- Failing to separate probe-side failures from target failures, so one bad region triggers alerts for thousands of customers
- Running the alert pipeline on the same infrastructure as the probes, so the outage that matters most also silences alerts
- Keeping every raw result forever in a relational database instead of downsampling into rollups
Likely follow-ups
- How would you avoid alerting thousands of customers when one of your probe regions loses connectivity?
- How would you make sure a check still runs if the scheduler node that owns it crashes mid-interval?
- How would you support multi-step checks such as logging in and adding an item to a cart?
- How would you compute an accurate monthly uptime percentage for an SLA report?
- How would you add SSL certificate expiry and DNS resolution checks?
- How would you deliver alerts to customer webhooks without a slow webhook delaying other alerts?
No community solutions yet
Be the first to publish your solution
Practice ‘Design Uptime Monitoring Service’ with an AI Interviewer
Get scored feedback on your diagram, scalability approach, and trade-offs. Free while we grow — up to 3 full interviews a day.