System Design Lab›System Design Questions›Design Metrics & Monitoring System
Design Metrics & Monitoring System
MediumInfrastructuretime-seriesstoragestreamingnotifications
Question Overview
Design a time-series monitoring platform that ingests millions of samples per second, powers dashboards, and fires alerts. The interesting parts are push versus pull collection, compressed time-series storage, downsampling, and keeping label cardinality from exploding.…
Sign up to see the full question and AI interviewer
Requirements
- Ingest counters, gauges, and histograms keyed by metric name plus labels, via both pull scraping and push
- Query with label filters and rate, sum, and percentile aggregations over any time range
- Dashboards refreshing every 30 s, with 24-hour queries returning in under 1 s
- Alert rules evaluated continuously, with deduplication, grouping, silencing, and routing to pager, chat, or email
- No dropped samples at peak, and data queryable within 30 s of emission
- Raw data for 15 days and 5-minute rollups for 1 year; alerting more available than the monitored systems
Back-of-the-envelope numbers
- Ingestion: 100M series ÷ 10 s interval = 10M samples/s; scrape-driven load is flat, so plan ~2× for catch-up after outages
- Raw volume: 10M samples/s × 16 B = 160 MB/s × 86,400 s ≈ 13.8 TB/day uncompressed
- Compressed: Gorilla-style encoding (~1.37 B/sample in Facebook's paper) → ≈ 13.7 MB/s ≈ 1.2 TB/day ≈ 18 TB for 15 days, ~53 TB with 3× replication
- Rollups: 288 five-minute points/day × 365 ≈ 105K points/series/year × 100M series ≈ 10.5T points; at ~8 B for min/max/sum/count ≈ 84 TB/year
- In-memory head: 2 hours = 720 samples × 100M series × 1.37 B ≈ 99 GB, plus ~1-2 KB/series of labels and index ≈ 100-200 GB
- Alert load: 50K rules evaluated every 30 s ≈ 1.7K rule queries/s, continuously
- Dashboard load: 5K engineers × 2 dashboards × 20 panels ÷ 30 s refresh ≈ 6.7K queries/s if all are open at peak
Key components
- Collectors: host agents or scrapers pull /metrics endpoints found via service discovery; a push gateway accepts batches from short-lived jobs
- Distributors: stateless ingest nodes that validate samples, enforce per-tenant series limits, and hash each series to ingesters with replication factor 3
- Ingesters: recent samples in an in-memory head block with a write-ahead log, Gorilla-compressed, flushed as 2-hour blocks to object storage
- Compression: delta-of-delta timestamps and XOR-encoded floats shrink 16-byte samples to under 2 bytes on typical data
- Inverted index: each label pair maps to a posting list of series IDs, intersected to resolve selectors such as service=api and status=500
- Compactor and query engine: build 5-minute and 1-hour rollups of min, max, sum, and count; queries auto-select resolution by time range
- Alerting: HA rule evaluators with for-durations to avoid flapping, plus an alert manager for dedup and routing, in a separate failure domain
Common mistakes
- Writing each sample as a row in a relational database, which cannot sustain 10M inserts/s or compress time series
- Putting unbounded values like user_id, request_id, or full URLs in labels, exploding series cardinality and ingester memory
- Averaging p99 latencies across hosts or rollup windows; percentiles must come from merged histograms or sketches
- Downsampling to averages only, which erases the short spikes that dashboards and alerts exist to catch
- Computing rates on counters without handling resets when a process restarts, producing large false negative dips
- Hosting alerting on the infrastructure it monitors with no dead man's switch, so an outage also silences its alerts
Likely follow-ups
- How would you compute an accurate p99 latency across 1,000 hosts?
- Would you choose push or pull for short-lived serverless functions, and why?
- How would you enforce per-team series limits without dropping critical metrics?
- How would you handle late or out-of-order samples that arrive after a network partition heals?
- How would you isolate a tenant whose cardinality explosion threatens the shared cluster?
- How would you keep alerting working through the loss of an entire region?
No community solutions yet
Be the first to publish your solution
Practice ‘Design Metrics & Monitoring System’ with an AI Interviewer
Get scored feedback on your diagram, scalability approach, and trade-offs. Free while we grow — up to 3 full interviews a day.