Design a Code Deployment System: Build, Artifacts, Rollout

Each server can end up running different, untested bytes, and one bad change reaches every customer at once with no quick way back. Deployment system design exists to stop exactly that.

Picture a common setup. A lending company runs about 60 services on 300 servers in two cloud regions, Mumbai and Hyderabad. Teams ship about 120 changes on a normal working day. The setup is made up, but the failure pattern is real.

At first, every team wrote its own deploy script. The payout service script logged into each of its 40 servers in turn. On each one it pulled the latest code, installed the Python libraries and restarted the app.

One Friday afternoon, a PDF library the service used published a new version. Staging, the test copy of production, still had the older version, installed on Monday. Production servers installed the new one during the evening deploy. Loan agreement PDFs began to fail, and without a signed agreement, no payout goes out.

All 40 servers had the change within two minutes. Rolling back took 40 minutes. The team rebuilt the old code, and the rebuild pulled the same new library again. In the end they pinned the old library version by hand.

That is the naive design: build on every server, update every server together. It failed in three separate ways.

• Nobody could say which bytes were running on which server.
• Staging had tested different bytes from production, installed on a different day.
• Nothing tried the change on a few servers first, so it hit every customer at once.

The rest of this guide designs a deploy platform for this company that fixes all three. Each piece comes from published practice, cited where it appears.

A code deployment system is the platform that turns a reviewed code change into a package and moves that package onto production servers safely, with a quick way back. It covers build, storage, rollout and rollback.

You need five words to follow the rest of this guide. Each gets a plain meaning and an example from the lending company.

Artifact. An artifact is the finished, ready-to-run package that a build produces. It can be a container image (a bundle of the app and everything it needs), a zip file or a binary. Example: the payout service built from commit 9f3c21a, one saved change in Git.

Digest. A digest is a fingerprint computed from every byte of a file with a hash function such as SHA-256. A hash function turns any data into a short, fixed-length code. Change one byte and the digest changes completely. Example: the payout artifact's digest starts with sha256:4be7.

Release. A release is one artifact plus the config for one environment, saved under its own number. Config means settings that differ by place, such as database addresses. Example: payout service release 418.

Rollout and wave. A rollout is the process of moving servers onto a new release. A wave is one step of it, a group of servers updated together. Example: wave 1 is a single payout server in Hyderabad.

Health gate. A health gate is an automatic check between waves that decides whether to continue, wait or roll back. Example: it compares error rates on new and old servers.

One picture helps here. Think of a theatre group touring a new play. Rehearsals are staging. The sealed final script is the artifact: no actor changes a line on the road. Preview shows in small towns are the first waves, before the big-city opening.

Where the picture breaks: a theatre group cannot take back a bad show, but a deploy platform can put the old release back in minutes. That undo is half of the design.

The requirements are to build every change into a fixed artifact, deploy any release to any environment, roll out in checked waves and roll back fast. It must also stay safe, keep an audit trail and scale.

First come the functional requirements: what the platform must do.

• Build each merged commit into one artifact and store it, named by its digest.
• Create a release from an artifact plus reviewed config, and never change it afterwards.
• Deploy a release to staging, then to production, in waves that the service team can set.
• Check health between waves, and roll back on its own when a gate fails.
• Let the release manager pause, stop or approve a rollout, and block deploys during freeze windows, set times when normal deploys are paused.
• Record who deployed what, where and when, so anyone can answer what is running.

Then come the non-functional requirements: the numbers and qualities it must hit. These are the company's own targets, not industry rules.

• Safety first: a bad release should reach under 5% of traffic before it is caught.
• Rollback in under 5 minutes, because payouts cannot wait for a rebuild.
• A full production rollout of one service in under an hour, including waiting time.
• Handle 120 deploys a day, with peaks of 20 an hour before lunch.
• If the platform itself goes down, running services keep running. New deploys and automatic rollbacks wait, so on-call staff need a manual way back to the last good release.

Rough maths: how big are the downloads?

Each artifact is about 250 MB. If all 300 servers pulled a new one at once from one store, that is 75 GB. A 1 Gbps link moves about 125 MB per second, so that takes 600 seconds.

Ten minutes of pure download is too slow for a 5-minute rollback. Two things help. A container image is stored as layers, stacked bundles of files, and Docker's docs say a layer already on the server is not downloaded again.

If only the app layer changes, say 30 MB, the same pull is 9 GB, or 72 seconds from one store. A cache in each region then keeps most of that traffic close to the servers, as a later section shows.

A build pipeline checks out one exact commit, builds it in a clean and locked environment, runs the tests, and saves the result in an artifact store under its digest. Only then can anything deploy it.

The key idea is a hermetic build. Google's Site Reliability Engineering (SRE) book says its builds are hermetic: insensitive to the libraries and other software installed on the build machine. Builds of the same code on different machines are expected to give identical results.

Lock the libraries, then check that it worked

For a Python service, that starts with a lock file. A lock file lists the exact version of every library, so a new release published upstream cannot sneak in. That alone would have stopped the Friday failure.

pip, Python's package installer, can go one step further with hash-checking mode. Each line pins a version and the SHA-256 hash of the exact file. Here is one line of such a requirements.txt file, using six, a small real library.

six==1.16.0 \
    --hash=sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254

Install with pip install --require-hashes -r requirements.txt. If a downloaded file's hash differs, pip stops with THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE and exit code 1, the signal for failure. Nothing is installed.

We checked this on pip 26.2.1 as of 19 October 2026. Any hash in the file switches the mode on by itself, and pip's message says so. Every library then needs a hash, including the libraries your libraries pull in, so teams generate the file with a tool rather than by hand. pip's secure installs page describes hash-checking mode as protection against remote tampering and network issues.

Here is the pipeline the platform team builds. Notice that nothing reaches the artifact store unless the tests pass.

Staging is the rehearsal. A rehearsal only helps if the actors rehearse the script they will perform. Before this design, staging rehearsed Monday's script while production performed Friday's.

The build record in the diagram has a formal name: provenance. SLSA, an open framework for software supply-chain security, uses it for a record of how an artifact was built: by whom, by what process, from which inputs.

SLSA specification v1.2 groups build safety into levels. At Build level 1, provenance exists, which catches mistakes but is easy to forge. At level 2, a hosted build platform generates and signs the provenance, adding a digital signature so any later change to the record shows.

At level 3, the platform also stops build runs from influencing one another and keeps signing keys away from build steps. The platform team aims for level 2 first.

You should build once because rebuilding for each environment means production runs bytes nobody tested. Build one artifact, test it in staging, and promote that exact artifact, checked by its digest, to production.

This is the sealed final script. Once the director seals it, every town sees the same play. If an actor wants a new line, the group rehearses a new version and seals that one instead.

The twelve-factor app method splits shipping into three stages. Build turns code into a runnable bundle. Release joins that bundle with the current config for one place. Run starts the app from a chosen release.

It adds two rules the platform follows. Every release has a unique ID, such as a timestamp or an incrementing number. Releases are an append-only ledger, so a release cannot change once created. Rolling back means choosing an older one.

Config lives apart from the artifact. One option Google's SRE book describes is two packages, one for the binary and one for its configuration. Here, config files sit in a reviewed repository, one folder per environment.

Names need care too. A tag, such as v42 or latest, is a label someone can push again to point at different bytes. Docker's docs call a digest an immutable identifier, and pulling by digest pins the exact image.

For a container image, the digest covers a small manifest that lists each layer's own digest, so it still pins every byte. That is why each release record stores the digest, never just a tag.

A digest check you can run

The agent on each server checks the digest of a zip or binary artifact before installing it. This sample writes a 3 MB stand-in artifact called artifact.bin in the current folder, checks it, then changes one byte and checks again.

import hashlib

def sha256_of(path):
    digest = hashlib.sha256()
    with open(path, "rb") as f:
        for block in iter(lambda: f.read(1 << 20), b""):
            digest.update(block)
    return digest.hexdigest()

def check_artifact(release, path):
    if sha256_of(path) != release["artifact_sha256"]:
        return f"release {release['id']}: digest mismatch, refusing to install"
    return f"release {release['id']}: digest matches, safe to install"

# A 3 MB stand-in for a real artifact
with open("artifact.bin", "wb") as f:
    f.write(b"x" * 3_000_000)
release = {"id": 418, "artifact_sha256": sha256_of("artifact.bin")}
print(release["artifact_sha256"][:16], len(release["artifact_sha256"]), "hex characters")
print(check_artifact(release, "artifact.bin"))

# Change one byte in the middle, as a bad download or tampering would
with open("artifact.bin", "r+b") as f:
    f.seek(1_500_000)
    f.write(b"y")
print(check_artifact(release, "artifact.bin"))

Run it with python3 on any recent Python. It prints this, and the first 16 characters are the same on every machine because the bytes are the same.

e55b8bdf621ddaa8 64 hex characters
release 418: digest matches, safe to install
release 418: digest mismatch, refusing to install

What just happened? The first check passed because the bytes matched the release record. One changed byte out of 3 million gave a completely different digest, so the second check refused.

Reading in 1 MiB blocks keeps memory small even for a large artifact. One MiB (mebibyte) is 1,048,576 bytes, 2 to the power 20, while the 3 MB stand-in is 3,000,000 bytes. To check a real file by hand on Linux, run sha256sum artifact.bin and compare its 64 hexadecimal characters (0 to 9 and a to f) with the release record.

A gradual rollout moves servers to the new release in waves of growing size. After each wave it waits, measures health, and only then starts the next one. A bad release stays small.

These are the preview shows. A touring group plays a few small towns first, where a flat scene costs little. By the big-city opening, the weak scenes are fixed.

Facebook's 2017 post on rapid release describes three tiers for its web code: employees first, then 2% of production, then 100%. Each release reaches all of production over a few hours, so the push can stop if problems appear.

Google's SRE book describes a similar shape. A rollout may start in one cluster and expand exponentially until all clusters are updated. For sensitive infrastructure, it may stretch over several days, interleaved across regions.

The payout service has 40 servers, 20 in each region. The platform team gives it this plan. The percentages assume the load balancer, which spreads requests across servers, does so evenly; the wait times are the company's choice.

1. 1

   Wave 1: one canary server

   One Hyderabad server gets release 418, about 2.5% of payout traffic. It bakes for 20 minutes. Bake time is the wait that lets slow bugs show.

2. 2

   Wave 2: a quarter of Hyderabad

   Five Hyderabad servers now run 418, about 12.5% of traffic. The gate checks again after 10 minutes.

3. 3

   Wave 3: all of Hyderabad

   All 20 Hyderabad servers move. Mumbai still runs release 417, so half the traffic stays on the old version.

4. 4

   Wave 4: all of Mumbai

   Mumbai moves only after Hyderabad has been healthy for 15 minutes. One region at a time means one bad region at most.

Inside a wave, traffic still has to move from old servers to new ones. You can do that with blue-green, canary or rolling updates. That choice has its own guide. The wave plan sits one level above it.

Here is the loop the orchestrator, the part that runs the plan, follows for each wave. Start at the top box and follow the Healthy arrow back. That arrow is what makes the rollout gradual.

Waves are not only for code. In July 2019, Cloudflare pushed one rule change for its web application firewall, which filters bad requests, to its whole network. A regular expression in it, a text-matching pattern, backtracked enormously: it kept retrying other ways to match, so the work exploded.

CPUs serving web traffic hit nearly 100%, and Cloudflare's report says the service went down for 27 minutes. A rule change went global in seconds, and its procedure allowed that. One listed fix was staged rollouts of rules, like its other software.

So this platform sends config releases through the same waves as code releases.

It mostly does not push. Each server runs a small agent that asks the control plane which release it should run, downloads that artifact from a nearby cache, installs it and reports back.

The control plane is the part of the platform that decides. It stores the desired state, meaning which release each group of servers should run. The agents do the work. They keep comparing what they run with what they should run, and fix any gap.

In this design, the pull model has three benefits. The deploy system needs no way to log in to servers. A server that was offline catches up on its own when it returns. If the control plane goes down, servers keep running their current release, though automatic rollback stops too.

Here is the whole system. Start at the console and follow the arrows to the agent. The agent never talks to the artifact store directly, only to its regional cache.

Agents poll the API, not the database. Polling every 15 seconds from 300 servers is 20 requests a second, a light load for one API service.

The regional caches fix the download maths from earlier. Each cache fetches a new 30 MB layer from the store once, so the store sends 60 MB instead of 9 GB. Each cache then serves its 150 servers: 4.5 GB, or 36 seconds at 1 Gbps.

A health gate compares the new servers with servers still on the old release, using a few clear signals. If the new release is clearly worse on any of them, the gate rolls back without waiting for a person.

Google's SRE workbook calls canarying effectively an A/B test: a comparison of two groups over the same period. It warns that time is one of the biggest sources of change in metrics, so before-and-after comparisons are hard to trust. Old and new servers at the same moment share the same lunchtime rush.

The workbook also says to keep absolute checks, such as service level objectives, the agreed targets for errors and speed. It suggests a few key metrics, perhaps no more than a dozen. The platform team picks four for the payout service.

• Error rate: the share of requests that fail.
• p99 latency: the time within which 99 of every 100 requests finish.
• Crashes: how often the app restarts on its own.
• Payouts per minute per server: the business number that shows whether money actually goes out.

Pause and predict. In the Friday failure, the payout request still returned success, and the PDF step failed later in the background. Which of the four signals would catch it?

A health gate you can run

Here is a simple gate with test cases. Each record holds numbers collected during the bake: one for the new servers, one for the old servers over the same minutes.

MIN_REQUESTS = 2000
MAX_ERROR_RATIO = 1.5    # new may have at most 1.5x the old error rate
MAX_LATENCY_RATIO = 1.2  # and at most 1.2x the old p99 latency
MIN_PAYOUT_RATIO = 0.8   # and at least 0.8x the old payouts per server

def gate(new, old):
    if new["requests"] < MIN_REQUESTS or old["requests"] < MIN_REQUESTS:
        return "wait"
    if new["crashes"] > 0:
        return "rollback: crashes"
    old_rate = max(old["errors"] / old["requests"], 0.001)
    new_rate = new["errors"] / new["requests"]
    if new_rate > old_rate * MAX_ERROR_RATIO:
        return "rollback: errors"
    if new["p99_ms"] > old["p99_ms"] * MAX_LATENCY_RATIO:
        return "rollback: latency"
    if new["payouts_per_min"] < old["payouts_per_min"] * MIN_PAYOUT_RATIO:
        return "rollback: payouts"
    return "continue"

old = {"requests": 40000, "errors": 120, "crashes": 0,
       "p99_ms": 310, "payouts_per_min": 10.0}
cases = {
    "too early": dict(old, requests=900, errors=3),
    "lunch rush": dict(old, requests=2100, errors=7, p99_ms=330),
    "one stray error": dict(old, requests=2100, errors=1),
    "PDF bug": dict(old, requests=2100, errors=6, payouts_per_min=6.0),
}
for name, new in cases.items():
    print(f"{name:16} {gate(new, old)}")

# The floor matters when the old servers had no errors at all
quiet_old = dict(old, errors=0)
print(f"{'quiet old fleet':16} {gate(cases['one stray error'], quiet_old)}")

Run it with python3. It prints this:

too early        wait
lunch rush       continue
one stray error  continue
PDF bug          rollback: payouts
quiet old fleet  continue

What just happened? Too early waits, because 900 requests prove little. The lunch rush continues: errors rose 1.11 times and p99 latency 1.06 times, both inside the limits. The PDF bug kept errors normal, but payouts fell from 10 to 6, so it rolled back.

That answers the prediction: only the business signal catches it. The last line shows why the old rate has a floor of 0.001. With zero old errors, one stray error in 2,100 requests would otherwise roll back a healthy release.

This is a teaching version with fixed ratios, which get noisy on small counts. Spinnaker, an open-source deploy tool, collects canary metrics with its Kayenta service. Its default judge then runs the Mann-Whitney U test, a statistical test that does not assume a bell curve.

As of 19 October 2026, Spinnaker's docs say the judge flags a metric only when it is 98% confident of a real difference. Whatever judge you use, keep a hard rule for crashes, as the sample does.

The last wave needs care. Once Mumbai moves, no old servers remain to compare with, so the gate there leans on the absolute checks, such as the service level objectives.

Rollback is just another deploy

The orchestrator sets the desired state back to release 417, and the agents do the rest. The old artifact is still in the cache, so there is nothing to build. This platform skips the bake time on the way back.

One thing can block a clean rollback: database changes. If release 418 renamed a column, release 417's queries still use the old name and fail. During a rollout both releases serve traffic at once, so the same rename breaks the 417 servers even before any rollback. The usual fix is to add new columns first and remove old ones in a later release.

The orchestrator saves every step of a rollout in the release database, so it can resume or reverse after a crash. A lock on each service makes sure only one rollout for it runs at a time.

Each rollout is a small state machine: a record that is always in exactly one named state. The states are pending, in wave 1, 2, 3 or 4, paused, rolling back, rolled back and done. If the orchestrator restarts, it reads the state and carries on.

The lock matters for the gate too. Google's SRE workbook strongly advises running only one canary deployment at a time. Two overlapping rollouts would leave the gate unable to tell which change moved the numbers.

The platform team adds a few more safety rules.

• Agents are idempotent: installing release 418 on a server that already runs 418 does nothing. A retried step is harmless.
• Freeze windows block normal deploys during busy times, such as month-end payout days. The release manager can approve an urgent fix.
• A stop button pauses every rollout at once, and anyone on call can press it. Facebook's post describes an emergency stop button for the same job.
• Every action goes into an append-only audit log: who, what, which release, when.

Now replay that Friday with the new platform. The lock file keeps the old PDF library, so the bug never ships. Suppose some other bug slips through. It reaches one server, about 2.5% of payouts.

Payouts per minute drop on that server, the gate fails, the orchestrator sets release 417 back, and each agent reinstalls it from the cache. Rollback takes minutes, not the 40 it took before.

Walk through the life of one change, from merge to the last server, and name the safety check at every step. Show that you care how many users a bad release can reach, not only how fast it ships.

1. 1

   Ask for the numbers

   Services, servers, regions, deploys a day and artifact size. Do the download maths out loud.

2. 2

   Draw the build

   Hermetic build from one commit with locked libraries, tests, then the artifact store keyed by digest, with a signed build record.

3. 3

   Define a release

   Artifact plus config, with a unique number, never edited. Say build once, promote everywhere.

4. 4

   Draw the control plane and agents

   Deploy API, release database, orchestrator, regional caches and pull-based agents that converge on a desired state.

5. 5

   Plan the waves

   One canary, then a slice, then a region, then the rest, with bake time between waves and config in the same waves.

6. 6

   Explain the gate and rollback

   Compare new with old at the same time, include a business signal, roll back automatically, and mention database changes and freeze windows.