System Design LabSystem Design QuestionsDesign LeetCode (Online Judge)

Design LeetCode (Online Judge)

MediumDistributed Systemsqueuesecuritysandboxingrankingscheduling

Question Overview

Design an online judge where users submit code in many languages and get a verdict against hidden test cases. The hard parts are running untrusted code safely in sandboxes, scaling a judge fleet for contest spikes, and keeping a live leaderboard accurate.…

Sign up to see the full question and AI interviewer

Requirements

  • Browse and search a catalog of problems by difficulty and tag, with statements and sample tests
  • Run code on sample tests or submit against hidden tests in 15+ languages with per-language limits
  • Return verdict, runtime, and memory, and keep a searchable submission history per user
  • Timed contests with a live leaderboard ranked by problems solved, then by penalty time
  • Untrusted code is fully isolated: no network, no host access, and bounded CPU, memory, processes, and disk
  • Verdicts within 5 s p95 normally and 30 s at contest peak; exactly one final verdict per submission

Back-of-the-envelope numbers

  • Submissions: 10M/day ÷ 86,400 s ≈ 116/s average, ~350/s at 3× daily peak
  • Judge CPU: 116/s × 2 CPU-s ≈ 230 cores busy on average, ~700 cores at 3× peak, before sandbox startup overhead
  • Contest burst: 100K participants × 10 runs and submits = 1M jobs ÷ 5,400 s ≈ 185/s average, ~550/s at 3× in the opening minutes
  • Contest CPU: 550/s × 2 CPU-s ≈ 1,100 extra cores, which must be provisioned before the contest starts, not reactively
  • Submission storage: 10M/day × ~2.2 KB (code plus metadata) ≈ 22 GB/day ≈ 8 TB/year
  • Test data: 3,000 problems × 50 tests × ~200 KB ≈ 30 GB, small enough to cache on every judge worker's local disk
  • Leaderboard: 100K participants × ~100 B ≈ 10 MB per contest, a single Redis sorted set

Key components

  • Submission API: stores the code and a QUEUED row, enqueues a job, and returns a submission_id; clients poll or receive the verdict over WebSocket
  • Durable job queue (Kafka or SQS) with separate contest and practice queues so practice traffic never starves a live contest
  • Judge workers: pull jobs under a lease, compile once, run each test in a sandbox, and write the verdict idempotently keyed by submission_id
  • Sandbox: gVisor or Firecracker microVMs, or containers hardened with seccomp, cgroup limits, no network, a read-only root, and per-test CPU timeouts
  • Warm pools: pre-started sandboxes per language so submissions do not pay container or VM startup time on the critical path
  • Test data store: versioned test sets in object storage, cached on worker disks, plus custom checkers for problems with multiple valid outputs
  • Leaderboard: a Redis sorted set per contest scored by solved count then penalty, persisted to the database and frozen near the end

Common mistakes

  • Executing submissions synchronously inside the web request, tying up API servers and collapsing the moment a contest starts
  • Using a default Docker container as the sandbox, sharing the host kernel with no seccomp profile or network isolation
  • Measuring runtime by wall-clock time on oversubscribed hosts, so the same code gets different verdicts from run to run
  • Acknowledging the queue message before judging finishes, so a crashed worker silently loses the submission
  • Autoscaling only after the contest starts, when new VMs take minutes to boot and the queue is already backed up
  • Computing the leaderboard with an ORDER BY over all participants on every refresh instead of maintaining it incrementally

Likely follow-ups

  • How would you stop a submission that forks thousands of processes or tries to fill the disk?
  • How would you keep runtime measurements fair across different worker hardware?
  • How would you rejudge every submission for a problem after fixing a wrong test case?
  • How would you support interactive problems where the program converses with a judge process?
  • How would you detect plagiarism between contest submissions?
  • What happens if an entire availability zone of judge workers fails mid-contest?

No community solutions yet

Be the first to publish your solution

Practice ‘Design LeetCode (Online Judge)’ with an AI Interviewer

Get scored feedback on your diagram, scalability approach, and trade-offs. Free while we grow — up to 3 full interviews a day.