System Design LabSystem Design QuestionsDesign Web Search Engine

Design Web Search Engine

HardSearchsearchrankingindexingdistributedcaching🔵 Google

Question Overview

Design a web-scale search engine that answers free-text queries over tens of billions of pages in under half a second. The hard parts are building and sharding an inverted index, fanning each query out to a thousand shards, ranking in stages, and keeping results fresh.…

Sign up to see the full question and AI interviewer

Requirements

  • Return the top 10 results with title, URL, and a query-specific snippet for any free-text query
  • Support phrase queries, pagination, spelling correction, and language or region filters
  • Rank by textual relevance, link-based page authority, and learned signals such as click feedback
  • Reindex changed pages within a day and make breaking news searchable within minutes
  • p99 under 500 ms; return partial results rather than errors when shards lag or fail
  • Suppress spam and near-duplicate pages so results stay diverse and trustworthy

Back-of-the-envelope numbers

  • Queries: 5B/day ÷ 86,400 s ≈ 58K QPS average, ~175K QPS at 3× peak
  • Index size: 20B docs × ~1,000 tokens = 20 trillion postings × ~1.5 bytes compressed ≈ 30 TB per full index copy
  • Sharding: 30 TB ÷ 1,000 document-partitioned shards ≈ 30 GB per shard, small enough to keep hot posting lists in memory
  • Fan-out: with ~50% of queries answered from the result cache, 87.5K QPS × 1,000 shards ≈ 87.5M shard lookups/s at peak
  • Leaf servers: 87.5M lookups/s ÷ ~2K lookups/s per server ≈ 44K servers, i.e. about 44 replicas of every shard
  • Document store for snippets: 20B × 10 KB ≈ 200 TB of text, ≈ 50 TB after ~4× compression
  • Indexing throughput: 1B reindexed docs/day ÷ 86,400 s ≈ 11.6K documents/s through parsing, dedup, and index building

Key components

  • Indexing pipeline: parse HTML, extract text and anchor text, detect language, drop near-duplicates with SimHash, then tokenize and build per-shard postings
  • Inverted index partitioned by document: each shard maps term → posting list of doc IDs, term frequencies, and positions, delta-encoded and compressed
  • Static quality scores such as PageRank computed offline over the link graph; postings ordered by quality so shards can terminate early
  • Query serving tree: the root parses and rewrites the query, fans out through aggregators to every shard, merges each shard's top-k, then fetches snippets
  • Multi-stage ranking: BM25 plus static score picks ~1,000 candidates per shard, then a learned-to-rank model reorders the merged top few hundred
  • Freshness tiers: a large base index rebuilt in batches plus a small real-time index for new and changed pages, merged at query time
  • Result cache keyed by normalized query and locale, plus hedged requests and per-shard deadlines to cut tail latency

Common mistakes

  • Sharding the index by term, which creates hot shards for common words and forces cross-shard intersection for multi-word queries
  • Running the expensive ranking model on every matching document instead of a cheap-to-expensive multi-stage funnel
  • Waiting for all 1,000 shards on every query so the slowest one sets latency, instead of using deadlines, hedging, and partial results
  • Rebuilding the whole index to achieve freshness instead of layering a small incremental index over the base index
  • Computing PageRank or other link signals at query time rather than offline as a static per-document score
  • Ignoring near-duplicate detection, which wastes index space and fills the results page with copies of the same content

Likely follow-ups

  • How would you compute PageRank over a 20B-page link graph and refresh it as the graph changes?
  • How would you make a breaking news article searchable within 60 seconds of publication?
  • How would you defend ranking against link farms and keyword-stuffed spam pages?
  • How would you add personalization or location signals without destroying the result cache hit rate?
  • How would you measure whether a ranking change actually improved result quality?
  • How would you roll out a new index build without a latency or relevance regression?

No community solutions yet

Be the first to publish your solution

Practice ‘Design Web Search Engine’ with an AI Interviewer

Get scored feedback on your diagram, scalability approach, and trade-offs. Free while we grow — up to 3 full interviews a day.