System Design Lab›System Design Questions›Design LeetCode (Online Judge)
Design LeetCode (Online Judge)
MediumDistributed Systemsqueuesecuritysandboxingrankingscheduling
Question Overview
Design an online judge where users submit code in many languages and get a verdict against hidden test cases. The hard parts are running untrusted code safely in sandboxes, scaling a judge fleet for contest spikes, and keeping a live leaderboard accurate.…
Sign up to see the full question and AI interviewer
Requirements
- Browse and search a catalog of problems by difficulty and tag, with statements and sample tests
- Run code on sample tests or submit against hidden tests in 15+ languages with per-language limits
- Return verdict, runtime, and memory, and keep a searchable submission history per user
- Timed contests with a live leaderboard ranked by problems solved, then by penalty time
- Untrusted code is fully isolated: no network, no host access, and bounded CPU, memory, processes, and disk
- Verdicts within 5 s p95 normally and 30 s at contest peak; exactly one final verdict per submission
Back-of-the-envelope numbers
- Submissions: 10M/day ÷ 86,400 s ≈ 116/s average, ~350/s at 3× daily peak
- Judge CPU: 116/s × 2 CPU-s ≈ 230 cores busy on average, ~700 cores at 3× peak, before sandbox startup overhead
- Contest burst: 100K participants × 10 runs and submits = 1M jobs ÷ 5,400 s ≈ 185/s average, ~550/s at 3× in the opening minutes
- Contest CPU: 550/s × 2 CPU-s ≈ 1,100 extra cores, which must be provisioned before the contest starts, not reactively
- Submission storage: 10M/day × ~2.2 KB (code plus metadata) ≈ 22 GB/day ≈ 8 TB/year
- Test data: 3,000 problems × 50 tests × ~200 KB ≈ 30 GB, small enough to cache on every judge worker's local disk
- Leaderboard: 100K participants × ~100 B ≈ 10 MB per contest, a single Redis sorted set
Key components
- Submission API: stores the code and a QUEUED row, enqueues a job, and returns a submission_id; clients poll or receive the verdict over WebSocket
- Durable job queue (Kafka or SQS) with separate contest and practice queues so practice traffic never starves a live contest
- Judge workers: pull jobs under a lease, compile once, run each test in a sandbox, and write the verdict idempotently keyed by submission_id
- Sandbox: gVisor or Firecracker microVMs, or containers hardened with seccomp, cgroup limits, no network, a read-only root, and per-test CPU timeouts
- Warm pools: pre-started sandboxes per language so submissions do not pay container or VM startup time on the critical path
- Test data store: versioned test sets in object storage, cached on worker disks, plus custom checkers for problems with multiple valid outputs
- Leaderboard: a Redis sorted set per contest scored by solved count then penalty, persisted to the database and frozen near the end
Common mistakes
- Executing submissions synchronously inside the web request, tying up API servers and collapsing the moment a contest starts
- Using a default Docker container as the sandbox, sharing the host kernel with no seccomp profile or network isolation
- Measuring runtime by wall-clock time on oversubscribed hosts, so the same code gets different verdicts from run to run
- Acknowledging the queue message before judging finishes, so a crashed worker silently loses the submission
- Autoscaling only after the contest starts, when new VMs take minutes to boot and the queue is already backed up
- Computing the leaderboard with an ORDER BY over all participants on every refresh instead of maintaining it incrementally
Likely follow-ups
- How would you stop a submission that forks thousands of processes or tries to fill the disk?
- How would you keep runtime measurements fair across different worker hardware?
- How would you rejudge every submission for a problem after fixing a wrong test case?
- How would you support interactive problems where the program converses with a judge process?
- How would you detect plagiarism between contest submissions?
- What happens if an entire availability zone of judge workers fails mid-contest?
No community solutions yet
Be the first to publish your solution
Practice ‘Design LeetCode (Online Judge)’ with an AI Interviewer
Get scored feedback on your diagram, scalability approach, and trade-offs. Free while we grow — up to 3 full interviews a day.