Design an Online Judge (LeetCode-Style) for Contest Scale

Three things go wrong. Strangers' code can crash the server or read the hidden answers, and the last minutes of a contest bring more submissions than one machine can run. Online judge system design is how you fix all three.

Picture a common setup. A competitive programming site runs a 90-minute contest every Sunday evening, with six problems and about 18,000 contestants. A small platform team, called the team below, runs the judge. The setup is made up, but the problems are real.

The first version was quick to build. The web app saved each submission, then compiled and ran it on the same 16-core server as an ordinary child process. It compared the output with the expected answer and saved the result.

This is the naive design, and one contest evening broke it in three ways. All the timings below are made up.

• A contestant submitted a program that kept copying itself, called a fork bomb. The server ran out of process slots and froze for 11 minutes.
• Another program opened the folder of hidden test files and printed the expected answers. It got Accepted on the hardest problem.
• In the last 15 minutes, 17,000 submissions arrived. Results that normally took 4 seconds took 22 minutes, and the leaderboard stopped moving.

The rest of this guide builds the judge that fixes all three. Strangers should not be able to break it, fool it or overwhelm it.

You need six words. Each one gets a plain meaning and an example from the contest site.

Submission. A submission is one piece of code a contestant sends for one problem, along with its language. Example: a C++ answer to problem D, sent at 9:12 pm.

Test case. A test case is one hidden input file plus its expected output. A problem on this site has 20 to 60 test cases, and contestants never see them.

Verdict. A verdict is the judge's result for a submission. Accepted means every test passed. Any other verdict names the first failure, such as Wrong Answer or Time Limit Exceeded.

Judge worker. A judge worker is a machine, or a process on one, that takes submissions from a queue, compiles them, runs them and reports verdicts. Workers do nothing else.

Sandbox. A sandbox is a locked-down space where a program runs with limited access. Inside it, the program cannot reach the network, other files or other programs, and it gets a fixed amount of memory.

Time limit. A time limit is the most time a program may spend on one test, such as 2 seconds. Go over it and you get Time Limit Exceeded.

Online judge system design is the design of a service that accepts code from anyone, runs it safely against hidden tests, and returns a correct, fair verdict quickly. LeetCode, Codeforces and DOMjudge, an open-source judge for programming contests, are online judges.

It looks like a normal web app, but it has three unusual needs. Every request carries untrusted code. Every request needs seconds of CPU, not milliseconds. And traffic arrives in sharp spikes at the end of each contest.

Picture a firecracker maker in Sivakasi testing samples from each new lot. Nobody lights a sample on the factory floor. Each one goes into a walled test pit, an inspector lights it, and a stopwatch times the burn.

An online judge works the same way. The sandbox is the walled pit, the time limit is the stopwatch, and the verdict is the inspector's note. The factory floor, meaning your web servers and database, never touches the code.

Where the picture breaks: a firecracker only explodes, but hostile code can also try quiet tricks, such as reading files it should not see. So the sandbox needs walls against snooping, not only against blasts.

Here are the main parts of the new design. Contestants' code only ever runs inside the sandbox box.

The judge queue is the key choice. A message queue is a list where one part leaves jobs and another part picks them up later. The API never waits for judging, and a spike becomes a longer list instead of a crashed server.

After you click Submit, the API saves your code, puts a job on a queue and replies at once with a submission id. A worker later picks up the job, judges it in a sandbox and writes back the verdict.

Follow one submission through the diagram below. Each box's detail line says which part talks to which.

The job on the queue is small: the submission id, the problem id and the language. The code itself stays in the database. If the queue ever loses jobs, the team can rebuild them from rows still marked Queued.

The browser learns the verdict by polling, which means asking the API every second or two, or through a push connection such as a WebSocket.

You run untrusted code safely by stacking several walls, because no single wall is enough. Use a fresh sandbox per run, no network, private files, an ordinary user, capped resources and filtered system calls.

Three Linux features build most of those walls. Each one is explained below in plain words.

Namespaces. A namespace makes a process see its own private copy of a system resource. A network namespace gives it its own network devices and ports. A mount namespace gives it its own view of the file system. A PID namespace gives it its own list of process ids. With a fresh /proc, the folder that lists processes, it cannot see other programs.

Control groups. A control group, or cgroup, puts processes into a group and shares out resources to that group. In cgroup version 2, memory.max is the hard memory limit for the group. Another file, pids.max, is a hard limit on the number of processes.

Seccomp. A system call is a request from a program to the operating system kernel, such as open a file or start a process. Seccomp limits which system calls a process may make. In filter mode, a small filter program checks each call against rules you write.

Here are the walls the new judge puts around every run:

• A new, empty sandbox for every run, deleted afterwards. Nothing from one contestant's run is left for the next one.
• No network. A fresh network namespace has only a loopback device, a pretend link that talks only to the same machine, so the program cannot download answers or reach other machines.
• Only the sandbox's own folder is writable. The worker copies in one input file at a time, and the expected output never enters the sandbox.
• An ordinary user id with no special rights, different for each sandbox, which helps keep two parallel runs out of each other's files.
• A small process limit. Once it is reached, any call that starts a new process fails, so a fork bomb hits the wall after a handful of copies.
• A system-call filter. Calls the judge never needs either fail or kill the process at once.

This also fixes the test-file cheat. The old judge ran code inside the folder that held every test file. In the new design, only the checker outside the sandbox ever reads the expected output.

You can check the network wall on any Linux machine as root. Run unshare --net ip -o link show. On our test machine (Linux 6.18) it printed one device, lo, in state DOWN, and a connect to a public address failed with Network is unreachable.

An online judge runs the tests one by one and reports the first test that fails, or Accepted if every test passes. Stopping at the first failure saves worker time, because one failure already decides the verdict.

DOMjudge makes stopping early an option called lazy_eval_results. Its manual says many submissions are expected to have some kind of error, so this significantly saves judging time. Without the option, DOMjudge runs every test and reports the highest-priority result.

The diagram shows the decisions for one submission. Start at the top and follow the arrows. Notice that only one path ends at Accepted.

Problems with more than one right answer

Some problems accept many outputs, such as any shortest path, or decimals within a small tolerance. For these, the problem setter writes a checker. A checker is a small trusted program that reads the input, the expected output and the contestant's output, then decides.

Beginners can stop here. You now know the whole path: queue, sandbox, limits and verdict. The next sections are for engineers: the exact limits and worker code, what isolate 2.7 really does, contest-night maths and fair timing.

An online judge enforces limits from outside the program. The sandbox measures CPU time and kills the program at a hard limit, and the kernel caps memory and processes. The program cannot switch these off, because it never controls them.

There are two kinds of time. CPU time is the time the processor actually spends running your program. Wall time is the time on a clock on the wall, from start to finish, including any waiting.

We measured both on a Linux 6.18 test machine. A child that slept for 1.5 seconds used 0.016 seconds of CPU time but 1.519 seconds of wall time. A busy loop of the same length used 1.370 CPU seconds and 1.525 wall seconds. Your numbers will differ by machine.

Judges mainly use CPU time, because it does not count time the processor spent on other work. DOMjudge, an open-source contest system, enforces limits primarily in CPU time. It adds a more lax wall-clock limit so a program that just sleeps cannot hold a worker.

In the Sivakasi pit, the inspector starts the stopwatch when the fuse catches, not when the sample joins the line. CPU time is that stopwatch. Wall time is the clock on the shed wall, which keeps running through every delay.

Soft and hard time limits

DOMjudge separates a soft limit from a hard one. The soft limit is the problem's real time limit. A setting called timelimit overshoot adds a margin to get the hard limit, and the program is killed only when the hard limit passes.

Isolate, an open-source sandbox built for programming contests, does the same with its --extra-time option. Its manual gives the reason: the judge can report the real run time, even when it is slightly over the limit. In isolate 2.7, --extra-time counts from the start of the run, not from the limit. So set it above --time, for example --time=2 with --extra-time=2.5.

The isolate 2.7 source code shows one more detail. After the program ends, it still reports status TO if CPU time went over the main limit, even when the program finished inside the extra time.

Memory limits

There are two common ways to cap memory, and they fail differently. An address-space limit, which isolate's --mem option sets, makes further allocations fail. In our test, Python raised MemoryError and the program kept running.

A cgroup memory cap works differently. When the group reaches memory.max and the kernel cannot free memory, it runs the out-of-memory killer, or OOM killer, inside that group. The killer ends a process.

In our test with a 64 MiB cap, a program touching 200 MiB was killed by SIGKILL, a signal that cannot be caught. Our machine only had cgroup version 1, so we used its memory.limit_in_bytes file, the older name for the same kind of cap. Isolate's manual says that in its control group mode, the kill can arrive as SIGSEGV instead. So decide MLE from cg-oom-killed, never from the signal number.

With control groups on, isolate 2.7 reads memory.peak, the group's highest memory use, which needs Linux 5.19 or newer. It writes cg-oom-killed into its result file when the group's memory.events shows an OOM kill.

Two smaller limits matter too. An output limit stops a program printing gigabytes onto the judge's disk. A file-size limit does the same for any file the program writes.

Turning a sandbox run into a verdict

After each run, isolate writes a meta file, one key:value pair per line. Its status codes are TO for timed out, RE for a non-zero exit, SG for died on a signal and XX for a sandbox error. A normal finish has no status line.

Here is how the worker turns one meta file and the program's output into a verdict for one test.

def read_meta(path):
    # isolate writes one "key:value" pair per line
    meta = {}
    with open(path) as f:
        for line in f:
            key, _, value = line.strip().partition(":")
            meta[key] = value
    return meta


def verdict_for(meta, output, expected):
    status = meta.get("status")      # missing when the run ended normally
    if status == "XX":
        return "IE"    # sandbox fault: rejudge, never blame the contestant
    if "cg-oom-killed" in meta:
        return "MLE"   # before RE: an OOM kill also ends with a signal
    if status == "TO":
        return "TLE"
    if status in ("RE", "SG"):
        return "RE"
    if not tokens_match(output, expected):
        return "WA"
    return "AC"


def tokens_match(output, expected):
    # ignore extra spaces and blank lines, compare word by word
    return output.split() == expected.split()

The order of the checks is the point. A sandbox fault comes first, so a judge bug never becomes a contestant's penalty. The OOM check comes before the crash check, because an OOM kill also ends the program with a signal.

The helper tokens_match compares outputs word by word, so a trailing space or an extra blank line never turns a right answer into Wrong Answer. We ran this code on nine sample results, and each gave the expected verdict.

The worker runs that check inside a loop over the tests. Here is that loop in Python. The helpers compile_in_sandbox and run_in_sandbox are stand-ins that call the sandbox. The first one returns None when compiling fails.

def judge(submission, tests, limits):
    binary = compile_in_sandbox(submission, limits)   # compiling is untrusted too
    if binary is None:
        return "CE", []
    results = []
    for test in tests:                         # small tests come first
        meta, output = run_in_sandbox(binary, test.input, limits)
        verdict = verdict_for(meta, output, test.expected)
        results.append((test.id, verdict, meta.get("time")))
        if verdict != "AC":
            return verdict, results            # stop at the first failure
    return "AC", results

Three lines matter. Compiling runs in the sandbox with its own limits, because the source file is untrusted input to the compiler. Tests are sorted small to large, so many wrong answers fail fast. The early return stops at the first failure.

We ran this loop with stand-in sandbox calls. A wrong answer on test 2 of 3 ran only two tests, and an accepted submission ran all three.

Isolate builds its sandbox from namespaces, a separate user per sandbox and resource limits, plus a short list of blocked system calls. It does not filter every call.

Its project page says several contest systems use it, most prominently CMS, the Contest Management System. As of 19 October 2026, its latest release is 2.7, from 24 August 2026.

In its 2.7 source code, each run gets new IPC, network, mount and PID namespaces. IPC (inter-process communication) namespaces separate shared memory and message queues between processes. The program then switches to an unprivileged user id that is different for each sandbox.

By default, isolate allows only one process. Its manual adds that time and memory limits do not work with several processes unless you turn on control group mode. In isolate 2.7, that mode needs cgroup version 2.

Here is the detail that surprises people. Isolate does not use a strict seccomp allow-list. Its manual says it places the program in namespaces instead, and blocks only a few calls, mostly ones that reach objects the kernel does not properly namespace.

Those blocked calls cover kernel keyrings (stores for keys and secrets), virtual machine sockets, file locks, io_uring (an asynchronous input and output interface) and 32-bit x86 programs on 64-bit machines. For keyrings, virtual machine sockets and file locks, the manual's reason is leaks between sandboxes. It blocks io_uring because it can create sockets of any address family, and 32-bit programs because they complicate call filtering.

The seccomp manual page recommends the opposite style where possible: an allow-list of expected calls, because a deny-list must be updated whenever a new dangerous call appears. Both views are reasonable for different jobs.

You handle the end-of-contest spike with a queue that absorbs the burst, enough workers started before the contest, and priorities so contest jobs never wait behind practice jobs. The queue turns the spike into waiting, not crashing.

Here is the maths, using the made-up contest numbers. 17,000 submissions in 900 seconds is about 19 per second. At about 4 CPU-seconds each, the judge needs about 76 CPU cores busy just to keep up.

Each worker machine has 8 cores. The team runs 6 sandboxes at a time on each, one per core, and leaves two cores for the worker program and the operating system. For 76 busy cores plus 50 percent headroom, that is about 19 machines.

On normal days, 3 machines handle practice traffic. Those 3 machines judge only about 4.5 submissions a second. After the 15-minute rush they would still have about 13,000 waiting, which takes about 48 more minutes to clear.

So the platform scales up to 20 machines 30 minutes before every contest starts. Waiting for an autoscaler, which adds machines as load rises, is too slow here, because a new worker needs a few minutes to boot and load its compilers (made up).

Inside the contest queue, fairness matters too. DOMjudge gives lower priority to submissions from teams that already have other submissions waiting, so one team cannot starve others. The team copies that idea and adds a rate limit: one submission per problem every 10 seconds per contestant.

The leaderboard is the other hot spot. Thousands of people refresh it in the final minutes. The team rebuilds it in a cache from new verdicts every few seconds, instead of querying the submissions table for every viewer.

You keep run times fair by making every judge machine identical and quiet: same hardware, same settings, one sandbox per core and nothing else running. A solution that takes 1.9 seconds should not pass on one worker and fail on another.

The isolate manual has a section on reproducible timings. The team applies its advice to every worker, and some of it can be checked with the isolate-check-environment program that ships with isolate:

• Turn off address space randomisation with sysctl kernel.randomize_va_space=0. The manual says it can affect timing, memory use and program behaviour.
• Set the CPU frequency governor to performance, and consider turning off frequency boosting, so the processor does not change speed mid-run.
• Run each sandbox on a single core, because the scheduler otherwise moves tasks between cores and pays cache costs. On CPUs that mix fast and efficient cores, pin to one kind.
• Turn off automatic transparent huge pages, a kernel memory feature. Turn off swap too, since memory limits do not cover data swapped out to disk.

For control group mode, the same manual says running isolate inside containers is not recommended. Container managers usually do not hand over control groups properly, and sharing the machine with other work would disturb timing.

Problem setters choose time limits from their own solutions. This site's rule is twice the slowest intended solution. If a test turns out wrong mid-contest, the setter fixes it and sends every affected submission to the rejudge queue.

Explain it as an ordinary web app wrapped around one dangerous part, the code runner. Spend most of your time on isolation, limits and the contest spike.

1. 1

   Ask the key questions

   Which languages, how many contestants, how many submissions per minute at peak, what time limits, and can outputs have more than one right answer?

2. 2

   Draw the background path

   API, database, queue, judge workers and test store. Say why the API replies at once instead of judging while the user waits.

3. 3

   Build the sandbox in layers

   Namespaces for network, files and processes, a process cap, cgroups for memory, a system-call filter, and separate machines with no secrets.

4. 4

   Define limits and verdicts

   CPU time with a looser wall-clock limit, a hard kill margin, memory through cgroups, and a fixed order for checking verdicts.

5. 5

   Size the contest peak

   Do the CPU maths out loud. Then explain scaling up before the contest, priority queues and per-user rate limits.

6. 6

   Cover fairness and recovery

   Identical tuned workers, judge jobs that are safe to repeat, rejudging after test fixes, and an internal error verdict that never blames the contestant.