Design a Distributed Task Scheduler: Cron at Scale

One cron server misses every job that falls due while it is down, and a second copy added for safety runs every job twice. Neither failure raises an error, so you usually hear about it from customers. A distributed task scheduler is built to fix both.

Picture a common setup. A reminder service sends bill, loan EMI and policy renewal reminders by SMS and email for banks and insurers. An EMI is the fixed monthly instalment you pay on a loan. The setup is made up, but the problem is real.

The service sends about 1.2 million reminders a day to 9 lakh (900,000) customers. The first version was one virtual machine running cron, the classic Unix tool that starts a command on a timetable. Every minute, cron started a script that sent the reminders due in that minute.

Cron reads its timetable from a file called a crontab, one line per job. The first five fields say when: minute, hour, day of the month, month and day of the week. So 0 9 5 * * means 9:00 a.m. on the 5th of every month, and a star means any value.

Then came two bad months. On an EMI due date, the machine rebooted for a security patch at 8:58 a.m. and took 49 minutes to come back. The script only ever looked at the current minute, so about 61,000 EMI reminders were silently skipped.

The team then added a second machine with the same crontab as a backup. Next month both machines fired every minute. About 1.1 lakh customers got every reminder twice, and the SMS bill for that day nearly doubled.

That is the whole problem: a timer on one machine gives you zero runs when it is down, and a copied timer gives you two. The rest of this guide replaces the crontab with a design where a crash causes a late run. A repeat is caught before it sends a second SMS, apart from a small gap between sending and recording, covered later.

You need five words: job, run, worker, lease and idempotent. Each one gets a plain meaning and an example from the reminder service.

• Job. A job is the standing instruction: what to do and on what timetable. Example: send customer 48213 an EMI reminder at 9 a.m. on the 5th of every month.
• Run. A run is one firing of a job at one scheduled time. The job above creates twelve runs a year, and each run gets its own id made from the job id plus its scheduled time.
• Worker. A worker is a process that does the actual work of a run. Here, a worker builds the message text and calls the SMS gateway, the outside service that delivers texts to phones.
• Lease. A lease is a claim on a run that expires by itself. A worker holds a run for 60 seconds, and if it neither finishes nor renews the claim in time, the run goes back for another worker.
• Idempotent. An action is idempotent when doing it twice has the same effect as doing it once. Marking a reminder as sent is idempotent. Sending an SMS is not, because the customer's phone buzzes twice.

A distributed task scheduler is a service that stores timed jobs and starts each run on time across a group of machines, surviving the loss of any one machine. Think of it as cron with a shared memory, many hands and written rules for failure.

Picture a city bus depot at 5 a.m. The timetable on the wall lists every departure. The dispatcher does not drive. She watches the clock, and when a departure is due, she sends the next free bus out on that route.

She also writes each departure in a logbook. A new dispatcher taking over at shift change can see which buses have left, so nobody sends the 5:10 bus twice.

Now map that onto software. The timetable is the job store, a database of jobs and their next run times. The dispatcher is the scheduler, which finds due jobs and hands them out. The buses are the workers, and the logbook is the run table.

Where the picture breaks: a dispatcher sees a bus leave. A scheduler never knows for sure that a worker finished, because a worker can die after sending and before reporting back. That gap is why the next sections need leases and duplicate checks.

A distributed task scheduler has five main parts: an API to create jobs, a job store, scheduler processes, queues of ready runs and a pool of workers. Monitoring watches all of them.

Follow a reminder from the bank's request to the SMS in the diagram. Notice that the scheduler never sends anything itself.

• Jobs API. Banks create, change and cancel reminders here. It saves each job with its first run time already worked out.
• Job store. One PostgreSQL database with a standby copy holds about 2.7 million jobs. In a local test, 2.7 million rows of the jobs table shown in the next section took about 280 MB with indexes. The average load is about 14 runs a second.
• Scheduler copies. Three identical processes look for due jobs every 2 seconds. If one dies, the other two carry on.
• Ready queues. Due runs wait here for a free worker. There is one queue per priority level.
• Workers. Twenty processes pull runs, send the message and record the result.
• Monitoring. It tracks scheduler lag, which is how many seconds after its scheduled time a run actually starts.

The scheduler keeps each job's next run time in an indexed column and asks, every few seconds, for rows whose time has passed. It never scans every job, and it never depends on catching one exact minute.

That fixes the first bad month. The old script asked only for reminders due this minute. The new query asks for anything due now or earlier, so after an outage the scheduler simply works through the backlog.

Here are the two tables. A timestamptz column in PostgreSQL stores an exact instant, so comparing it with now() works the same from any server. The partial index covers only active jobs, so finished jobs cost the query nothing.

CREATE TABLE jobs (
  id          bigint PRIMARY KEY,
  cron        text NOT NULL,          -- for example '0 9 5 * *'
  tz          text NOT NULL,          -- the customer's zone, 'Asia/Kolkata'
  next_run_at timestamptz NOT NULL,   -- an exact instant, compared in UTC
  priority    text NOT NULL,          -- 'urgent', 'normal' or 'bulk'
  active      boolean NOT NULL DEFAULT true
);
CREATE INDEX jobs_due ON jobs (next_run_at) WHERE active;

CREATE TABLE runs (
  run_id    text PRIMARY KEY,         -- job id + scheduled time
  job_id    bigint NOT NULL,
  status    text NOT NULL DEFAULT 'queued',
  queued_at timestamptz NOT NULL DEFAULT now()
);

When a run is queued, the scheduler works out the next run time from the cron expression and saves it. A job for the 5th of every month gets 5 November as soon as the 5 October run is queued.

Read the timetable on the customer's clock, then store the result as an exact instant. UTC is the world reference clock, with no daylight saving. A 9 a.m. reminder in Asia/Kolkata is always 03:30 UTC, but in New York 9 a.m. moves between 13:00 and 14:00 UTC.

Sharing the work without clashing

Three scheduler copies run the same query. Without care, all three grab the same job and queue it three times, which is the second bad month again.

There are two common fixes. One is to elect a single leader that does all the launching while the other copies wait as standbys. Google's SRE book describes its distributed cron this way: replicas agree through Paxos, a method that lets machines settle on one value even when some fail. Only the leader changes the shared state.

The other fix lets every copy work and lets the database hand out rows. In PostgreSQL, FOR UPDATE SKIP LOCKED locks the rows a query picks and makes other copies skip them instead of waiting.

The PostgreSQL docs say this view of the data is inconsistent, so it is not for general work. They name the exact use: many consumers reading a queue-like table. The reminder service already runs PostgreSQL, so the team picks row locks.

The scheduler loop in Python

Here is one tick of the scheduler. It uses psycopg 3.3.6, a PostgreSQL driver, and croniter 6.2.4, a library that works out the next time a cron expression matches. Pass a psycopg connection opened with autocommit=True, as in our test. The send function stands in for your queue client.

from datetime import datetime
from zoneinfo import ZoneInfo
from croniter import croniter

DUE_SQL = """
    SELECT id, cron, tz, next_run_at, priority FROM jobs
    WHERE next_run_at <= now() AND active
    ORDER BY next_run_at LIMIT 500
    FOR UPDATE SKIP LOCKED
"""

def next_time(cron, tz, after):
    local = after.astimezone(ZoneInfo(tz))     # read the timetable on the customer's clock
    return croniter(cron, local).get_next(datetime)

def scheduler_tick(conn, send):
    with conn.transaction():
        for job_id, cron, tz, due, priority in conn.execute(DUE_SQL).fetchall():
            run_id = f"{job_id}:{int(due.timestamp())}"   # same run, same id
            conn.execute("INSERT INTO runs (run_id, job_id) VALUES (%s, %s) "
                         "ON CONFLICT (run_id) DO NOTHING", (run_id, job_id))
            send(run_id, priority)
            conn.execute("UPDATE jobs SET next_run_at = %s WHERE id = %s",
                         (next_time(cron, tz, due), job_id))

Three lines matter. FOR UPDATE SKIP LOCKED gives each copy its own rows. The run id joins the job id with the scheduled time in Unix seconds, so the same run always gets the same id. ON CONFLICT DO NOTHING makes a second insert of that id quietly do nothing.

We ran this code on PostgreSQL 16.15 with three copies against 10,000 due jobs, 17 times. Each time the queue got exactly 10,000 sends, no run id repeated and no job was left due. The split changed from run to run: each copy took between 2,500 and 4,500 jobs.

Then we removed FOR UPDATE SKIP LOCKED and ran it 11 times. The queue got about twice as many sends for the same 10,000 jobs: between 20,000 and 22,000, depending on timing. That is the backup-crontab month again, now in code.

Check it yourself: in a test, make send add each run id to a list in each copy. Then compare the total count with the count of distinct ids, and run SELECT count(*) FROM jobs WHERE next_run_at <= now(). Expect equal counts and zero still due.

One gap remains. A commit is the step that makes database changes permanent. If it fails after send, the new next run time is not saved, and the next tick queues the same run again. Our test showed both messages carry the same id, 1:1791171000, so the worker's duplicate check catches the repeat.

The backlog has a catch, too. A job that fires every minute and missed 49 minutes is not collapsed into one run. In our test, the loop queued 50 late runs, one per tick, until it caught up.

You stop missed runs by saving state before you act and by using leases. You make double runs harmless by giving every run a stable id and checking it before any action. The scheduler aims for at least once, and duplicate checks do the rest.

Mature tools say the same. The Kubernetes docs say a CronJob creates a Job approximately once per scheduled time. In certain circumstances two Jobs might be created, or none. Their advice is to make your Jobs idempotent.

Here is the life of one run. Read it top to bottom and watch the lease. It turns a crashed worker into a short delay instead of a lost reminder.

Queues such as Amazon SQS build the lease in and call it a visibility timeout. A received message stays in the queue but is hidden, for 30 seconds by default, as of 19 October 2026. If the consumer does not delete it in time, it becomes visible again.

That design means a slow worker can cause a duplicate. If sending takes 70 seconds and the lease is 60, a second worker may take the same run. The fix is a lease that fits the work, extended by a heartbeat while the worker is busy.

In SQS, the heartbeat is a ChangeMessageVisibility call. The docs cap the total at 12 hours from the first receive, and extending does not reset that cap. They also say a standard queue can deliver a message more than once.

The duplicate check

Before sending, the worker looks up the run id in the run table. If the run is already marked sent, the worker skips the SMS and acknowledges the message, which tells the queue to delete it.

A small gap is left between sending and recording. If the worker dies there, the next worker sends again. If your gateway accepts a reference id and drops repeats it has already seen, pass the run id, as the handler below does. Check your gateway's docs before you rely on this.

One trap hides in the timing. The scheduler sends inside its transaction, so a fast worker can finish before the run row is committed. A plain UPDATE that sets status to sent then finds no row.

In our test, that UPDATE changed 0 rows and the run stayed queued, so a later repeat would send again. Write mark_sent as an upsert, an insert that updates the row if it already exists:

-- mark_sent as an upsert: waits for the scheduler's insert to commit, then marks it
INSERT INTO runs (run_id, job_id, status) VALUES (%s, %s, 'sent')
ON CONFLICT (run_id) DO UPDATE SET status = 'sent';

Run against the same race, the upsert waited for the scheduler's commit, 1 second in our test, then marked the run sent. Write mark_failed the same way, with failed in place of sent. In the same race, a plain UPDATE to failed also changed 0 rows, while the upsert recorded the failure.

What happens to runs missed during an outage?

When the scheduler returns after downtime, it finds every overdue job at once. You need a rule for each kind: run it late, or skip it.

Kubernetes has a field for this, startingDeadlineSeconds. A run that cannot start within that many seconds is skipped, and Kubernetes treats it as a failed Job. The docs warn that values under 10 seconds may stop the CronJob being scheduled at all.

The team writes its rules down per reminder type:

• EMI due today: send late, up to 8 p.m. the same day. A late reminder can still save a late fee.
• Bill due in three days: always send late. The due date is still ahead, so the reminder still helps.
• Renewal offer that closes at noon: skip it after noon. A reminder for a closed offer only annoys people.

A task scheduler should retry only temporary failures, wait longer after each attempt with some randomness added, and stop after a fixed number of attempts. Runs that still fail go to a dead-letter queue, and permanent failures are never retried.

Back at the bus depot, a bus breaks down halfway along its route. The dispatcher sends a spare bus to finish the trip, and the passengers arrive a little late. But if the road itself is closed, sending ten spare buses down it helps nobody.

Software failures split the same way. A temporary failure may go away on its own, like a gateway timeout or a 503 (service unavailable). A permanent failure never will, like a nine-digit phone number or a customer who opted out.

Waiting longer each time is called exponential backoff. Adding randomness to each wait is called jitter. Without jitter, 50,000 runs that failed together at 9:00 would all retry in the same second and knock the gateway over again.

Here is the worker's handler. The db and sms objects stand in for your own code. It returns what the queue should do next, acknowledge, retry or dead-letter, plus a delay in seconds.

import random

MAX_ATTEMPTS = 6                  # the first try plus five retries

class TemporaryError(Exception): pass   # 503, or a timeout if the gateway drops repeated refs
class PermanentError(Exception): pass   # invalid number, opted out

def handle(run, db, sms):
    if db.is_sent(run.run_id):                   # duplicate delivery
        return ("ack", 0)
    try:
        sms.send(run.phone, run.text, ref=run.run_id)
    except PermanentError:
        db.mark_failed(run.run_id)
        return ("ack", 0)
    except TemporaryError:
        if run.attempt >= MAX_ATTEMPTS:
            return ("dead_letter", 0)
        cap = min(600, 30 * 2 ** (run.attempt - 1))   # 30, 60, 120, 240, 480 s
        return ("retry", random.uniform(0, cap))
    db.mark_sent(run.run_id)                     # job_id is the part before the colon
    return ("ack", 0)

The first check is the duplicate check from the last section. The PermanentError branch acknowledges the run without a retry, because no retry fixes a wrong number.

The cap doubles with each attempt: 30, 60, 120, 240 and then 480 seconds. random.uniform picks a wait anywhere below the cap, which the AWS Architecture Blog calls Full Jitter. The five waits add up to at most 930 seconds, about 15 minutes.

We ran the handler with 50,000 runs failing together on their first attempt. Without jitter, all 50,000 retries land in one second. With it, they spread over 30 seconds, and the busiest second got 1,750.

That is still above a gateway limit of 500 a second, so jitter alone is not enough. A rate limiter in front of the gateway does the rest, as the spike section shows.

Be careful with timeouts. A timeout does not tell you whether the SMS went out. The retry may send a second text, so a timeout is only safe to retry when the gateway drops repeats of the same reference id.

After six attempts, the run goes to the dead-letter queue, a separate queue for runs that failed too often. Nothing there retries by itself. The on-call team gets an alert, fixes the cause and replays what still matters.

Priorities let urgent runs reach a worker first, usually through separate queues that workers read in a weighted order. The hard part is not making urgent work fast. It is making sure low-priority work still gets done.

The reminder service has three levels. Urgent runs warn about a payment that bounced this morning. Normal runs are bill and EMI reminders. Bulk runs are renewal notices weeks ahead, which nobody minds getting at 9:20 instead of 9:00.

The simplest scheme, strict priority, always serves urgent first. Its danger is starvation: if urgent work never stops, bulk work never runs. The team uses weighted reading instead.

Of every 10 runs a worker takes, 6 come from urgent, 3 from normal and 1 from bulk. When one queue is empty, the worker borrows from the next.

The team also reserves 4 of the 20 workers for the urgent queue alone, so a busy morning never traps an urgent run behind bulk work. The table compares the common schemes.

Airflow, a popular open-source workflow scheduler, shows why you should read the fine print. As of Airflow 3.3.2, each task's priority_weight is 1 by default, and higher numbers are queued first.

But the default weight rule, called downstream, adds the weights of every task after it. The source code computes the task's own weight plus the sum of all downstream weights. Set weight_rule to absolute if you want the number you typed.

You spread runs that do not need an exact minute across a window, and you cap how fast workers call outside services. Otherwise everyone's favourite time, like 9:00 a.m., becomes your outage time.

This spike is called a thundering herd: many things wake at the same moment and rush one resource. Here, about 40% of all reminders, roughly 480,000, are set for 9:00 a.m. The SMS gateway accepts 500 messages a second, so even perfectly spread, that is 16 minutes of sending.

Google's SRE book describes the fix it uses. Its crontab format allows a question mark in a field, meaning any value is acceptable. The value is then picked by hashing the cron job's configuration over the allowed range. Hashing turns text into a number that looks random but never changes for the same input, so launches spread out.

The reminder service does the same for customers who chose a morning reminder. A hash of the customer id picks a fixed minute between 9:00 and 9:29. The same customer always gets the same minute, so the reminders still feel regular.

We checked the spread with a SHA-256 hash of 480,000 customer ids over 30 minutes. Every minute got between 15,798 and 16,297 reminders. At 450 a second, the busiest minute needs about 36 seconds of sending.

That 450 comes from a rate limiter in front of the gateway, set a little under its limit. Workers that hit the cap wait instead of collecting errors. Urgent runs skip the morning spread.

Time zones break a scheduler in two places: the clock change itself and the way you build the run id. Both slip past the duplicate check, and both showed up in our tests.

India has no daylight saving, but customers abroad do, and that is where the stable run id stops helping. We ran next_time for New York jobs across the 2026 clock changes on croniter 6.2.4.

On 1 November 2026, clocks go back and 1:30 a.m. happens twice. For a 30 1 * * * job, next_time returned 1:30 a.m. twice, at 05:30 and 06:30 UTC. Those are two different instants, so the run ids differ and the duplicate check lets both through.

On 8 March 2026, clocks went forward and 2:30 a.m. never happened. A 30 2 * * * job ran at 3:00 a.m. instead of being skipped. The croniter README says only to pass a time-zone-aware datetime, so treat both behaviours as undocumented as of 19 October 2026 and pin your version.

If you want one run per local time, build the run id from the local wall-clock time, such as 7:2026-11-01T01:30. Both fall-back fires then share an id, and the duplicate check drops the second.

A second time-zone bug hides in the run id itself. A tempting way to build it is from the timestamp's ISO text. psycopg returns the timestamp in each session's time zone, so the same run printed as 03:30+00:00 in one copy and 09:00+05:30 in another.

Two copies with different session settings would then insert two different ids for one run. Unix seconds, as in the code above, gave 1791171000 in all three sessions we tried.

Explain it as cron plus four answers: where jobs live, who decides a run is due, who does the work and what happens on failure. Use one example throughout.

1. 1

   Clarify the scale and the cost of mistakes

   Ask how many jobs exist, how many runs fire at peak and how late a run may start. 1.2 million runs a day sounds big but averages about 14 a second.

2. 2

   Store jobs with their next run time

   Describe a jobs table indexed on next run time, in UTC. Add a run table keyed by job id plus scheduled time.

3. 3

   Split the scheduler from the workers

   Several scheduler copies claim due jobs with row locks, or one elected leader does it. They only queue runs. Workers do the sending.

4. 4

   Make every run safe to repeat

   Say that delivery is at least once. Give each run a stable id, check it before acting, and use leases so a crashed worker delays a run instead of losing it.

5. 5

   Add retries and priorities

   Retry temporary errors with backoff and jitter, and dead-letter after a cap. Use weighted queues or reserved workers for urgent jobs, and name starvation.

6. 6

   Cover spikes and monitoring

   Spread flexible jobs across a window and rate-limit calls to outside services. Alert on scheduler lag and on dead-letter queue growth.