Deployment Strategies: Blue-Green, Canary and Rolling

When every user gets new code at the same moment, one hidden bug reaches all of them, and you find out from your users. If going back is slow, they keep hitting it until you finish.

Picture a common setup. A small-loans app lets shopkeepers and delivery riders borrow between Rs 5,000 and Rs 50,000 for a few weeks. It takes about 18,000 loan applications a day on 8 servers. The setup is made up, but the failure is a classic one.

The team ships every Thursday evening. Release 41 changed how interest is rounded on the repayment schedule, and every test passed. The release engineer stopped all 8 servers, copied the new code on and started them again, with 4 minutes of downtime.

Ten minutes later, approvals began to fail. Applicants who chose a 45-day loan, about 15 percent of them, hit a rounding bug that only real data triggered. The test cases had used 30 and 60 days.

Going back was slow. Nobody had kept the old build ready, so release 40 had to be rebuilt and copied to every server. Approvals stayed broken for 50 minutes, and about 600 applicants gave up.

The review asked two questions. Why did every user get the new code at once? And why did going back take longer than going forward? Rolling, blue-green and canary deployments each answer them differently.

A deployment strategy is your plan for replacing the running version of an app with a new one. It decides who sees the new code, in what order, and how you go back. Five words carry the rest of this guide.

Deploy and release. To deploy is to put new code on servers. To release is to let users reach it. Good strategies keep these two steps apart.

Load balancer. A load balancer receives every request and passes it to one of your servers. Changing its settings is how most strategies move users from one version to another.

Health check. A health check is a small test request sent to each server again and again, asking whether it is ready. A server that fails it stops getting users.

Rollback. A rollback means going back to the last good version. How fast you can roll back decides how long users suffer from a bad release.

Recreate. Release 41 used the simplest strategy: stop everything, then start the new version. Kubernetes, a widely used system for running apps in containers across many machines, offers this as its Recreate strategy.

Its documentation says all existing copies are killed before new ones are created. That gives two problems. Every release has downtime, even a good one, and every user meets a hidden bug at the same moment.

A rolling deployment replaces old copies of your app with new ones a few at a time. Old and new copies serve users side by side until every copy is new, so the app never fully goes down.

In Kubernetes, each running copy of your app is called a pod. A Deployment is the object that says how many pods you want and which version they run. RollingUpdate is its default strategy.

Two settings control the pace. The maxSurge setting is how many extra pods may run above your normal count during the update. The maxUnavailable setting is how many of your normal count may be missing or not ready.

Kubernetes counts a new pod only once it passes its health check. With maxUnavailable set to 0, no old pod goes until a new one is ready; with the defaults, up to a quarter of the old pods can go first. Either way, a version that never passes stalls instead of replacing everything. Going back means another rolling update, which takes minutes, not seconds.

Blue-green deployment runs two full copies of production, called blue and green. One serves every user while you install and test the new version on the other. Then you switch all traffic across in one step.

Think of a city opening a new flyover. Engineers build it beside the old road while traffic keeps moving below. Inspectors test every joint. On opening day, all traffic goes up at once, and the old road is still there.

Where the picture breaks: a city can send cars back to the old road, but a release often shares one database with the old version. Data written by the new code does not go away when you switch back.

Martin Fowler describes the same idea. The two environments should be as identical as possible, and a router decides which one is live. To roll back, you switch the router back to the old environment, which is still running.

Here is the setup during release 42. Follow the arrows out of the load balancer. Notice that both colours share one database, which is where the hard part hides.

A blue-green release now runs in four moves. The release engineer deploys release 42 to green while blue serves users. The QA lead runs smoke tests, a short set of checks on the most important paths, against green.

Then the load balancer flips, and every new request lands on green. Blue stays up and untouched for 24 hours. If approvals fail, flipping back takes seconds, because nothing needs to be rebuilt or started.

Blue-green has real costs. Google's SRE workbook, a book on site reliability engineering, says the setup uses twice as many resources as a traditional deployment. In the cloud you can create green only for the release and delete the old colour afterwards, which keeps the doubling short.

The deeper catch is that the switch is still all at once. The 45-day rounding bug would hit every 45-day applicant the moment green went live. Blue-green makes recovery fast, but it does not shrink how many users see a bad release.

A canary deployment sends the new version to a small share of users first. It compares how they fare against users on the old version, and widens the share only while the numbers stay healthy.

The name comes from coal mining. Miners carried a canary in a cage underground. If toxic gas leaked in, the bird died before the miners did, which gave them time to get out. A canary release is the same early warning for production.

The two groups have names. The canary is the slice on the new version. The control is everyone else, on the old version, at the same time. The SRE workbook warns against comparing with yesterday instead, because traffic differs, such as weekdays against weekends.

A small slice limits the damage. The SRE workbook gives an example: a 20 percent error rate on a 5 percent canary is a 1 percent error rate overall. For the loan app, a 5 percent canary means about 1 in 20 applicants meets release 42.

Here is the loop the team sets up for the loan service. Start at the top and follow the arrows round. The decision box is where release 41 would have been stopped.

Danilo Sato's article on Martin Fowler's site lists common choices: a random sample, your own staff first, users chosen by profile, or one region first. The team starts with staff accounts, then picks applicants by a hash of their user id.

A hash turns the id into a fixed number. The same applicant then always lands on the same version, instead of seeing release 41 on one screen and release 42 on the next.

Blue-green moves every user to the new version in one switch and relies on switching back fast. A canary moves a small slice first and relies on comparing that slice with everyone else before going further.

So blue-green cuts recovery time, while a canary cuts the number of users a bad release can reach. The table puts all four strategies side by side on the loan service.

Why does the admin portal get blue-green and not a canary? Only 40 staff use it. A 5 percent canary would be two people, which is far too little traffic to judge. A full test on green before the switch tells the QA lead much more.

The two strategies also mix. The SRE workbook describes putting the new version on the idle colour and slowly shifting traffic to it. The idle colour becomes the canary and the live one the control. That needs a weighted split on the load balancer, not a single switch.

Kubernetes rolls a Deployment forward and back with the same rolling update, paced by two settings. Neither direction is instant, and nothing rolls back on its own.

What the rollout settings do

The maxSurge and maxUnavailable settings both default to 25 percent. Kubernetes rounds surge up and unavailable down, so on 8 pods the defaults allow 10 pods in total and as few as 6 ready ones. The team sets both by hand in its Deployment file, written in YAML, a plain-text settings format.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: loan-api
spec:
  replicas: 8
  revisionHistoryLimit: 10      # old versions kept for rollback
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 2               # up to 2 extra pods while updating
      maxUnavailable: 0         # keep 8 ready pods during the rollout
  selector:
    matchLabels: {app: loan-api}
  template:
    metadata:
      labels: {app: loan-api}
    spec:
      containers:
        - name: api
          image: "registry.example.com/loan-api:42"
          readinessProbe:       # the health check that gates the rollout
            httpGet: {path: /healthz, port: 8080}
            periodSeconds: 5

The maxSurge line lets Kubernetes start 2 new pods before removing any old ones. The maxUnavailable line means the rollout never removes an old pod if that would leave fewer than 8 ready ones. The revisionHistoryLimit line keeps the last 10 old versions on record, which is also the default.

The readinessProbe lines are the health check. Every 5 seconds, Kubernetes asks the pod for /healthz on port 8080, and only a passing answer marks it ready.

With maxUnavailable: 0, only a ready new pod lets Kubernetes remove an old one. With these settings, 8 pods change in four waves of 2. Leave the probe out and Kubernetes treats the check as passed, as the probes docs say. A broken build that starts cleanly then sails through.

Pause and predict: what if release 42 cannot start?

Suppose release 42 starts but never passes its readiness check. Does Kubernetes replace all 8 pods anyway? Think about what maxUnavailable: 0 allows before reading on.

It does not. A small simulation of the v1.37.0 controller logic, not a live cluster, shows 2 new pods stuck and all 8 old pods still serving. With the 25 percent defaults, the same failure leaves 6 old pods serving and 4 stuck new ones, so capacity drops by a quarter.

Kubernetes then waits. After the progress deadline, 600 seconds by default, it marks the Deployment as failing to progress. The Deployments page says it takes no other action, so a person or a tool must roll back.

That is narrower than the feature summary on the kubernetes.io home page, which says Kubernetes will roll the change back for you. As of 19 October 2026, the Deployments page and the v1.37.0 source agree: no automatic rollback.

How to roll back and check it worked

Rolling back is one command in kubectl, the Kubernetes command-line tool: kubectl rollout undo deployment/loan-api. Going by the docs example, it prints deployment.apps/loan-api rolled back.

The v1.37.0 kubectl source shows what undo does. It copies the old version's pod template, the part that says what each pod runs, back into the Deployment. The controller then finds the old ReplicaSet, the object that kept that version's pods, and scales it up.

So rollback is another rolling update, at the same pace as the release. If each pod takes about 60 seconds to become ready (made up), 8 pods in waves of 2 take about 4 minutes. The bug keeps reaching users during those minutes.

To check either direction, run kubectl rollout status deployment/loan-api. It waits, then prints deployment "loan-api" successfully rolled out and exits with code 0. A stuck rollout prints waiting lines. Once 600 seconds pass with no progress, it prints error: deployment "loan-api" exceeded its progress deadline and exits with code 1.

A trustworthy canary keeps each user on one version, compares the canary with the control at the same time, and waits for enough data before it decides. Each part below was run in Python 3.13.

Keep each user on one version

Here is the bucketing by user id from the canary section, in Python.

import hashlib

def bucket(user_id):
    digest = hashlib.sha256(user_id.encode()).digest()
    return int.from_bytes(digest[:8], "big") % 100   # 0 to 99

def on_canary(user_id, percent):
    return bucket(user_id) < percent

print(bucket("applicant-1042"), on_canary("applicant-1042", 5))

Run it and it prints 64 False every time: this applicant is in bucket 64, so outside a 5 percent canary. Across 100,000 test ids, 4.98 percent landed under 5. Raising the share to 25 keeps every user already on the canary.

Do not use Python's built-in hash() for this. By default it is randomised for strings in every new process, so three runs on Python 3.13 gave 86, 67 and 26 for the same id. A fixed function such as SHA-256 gives the same bucket on every server.

Kubernetes alone does not do this. Its docs show a canary as a second Deployment behind the same Service, the stable address that spreads requests across pods with matching labels. Pod counts set the split, such as 3 stable to 1 canary, so traffic splits across pods, not by user id. Sticky user buckets need a routing layer that reads the id.

The canary check in Python

Here is the check that decides each step. The canary and control objects stand for numbers your monitoring already collects per group: requests, errors, p99 latency and applications. The p99 latency is the time within which 99 percent of requests finish.

MIN_REQUESTS = 2000        # fewer requests than this tell you little
MIN_APPLICATIONS = 400     # approvals are noisier, so wait for more
MAX_ERROR_RATIO = 1.5      # canary errors may be at most 1.5x control
MAX_EXTRA_P99_MS = 150     # canary p99 may be at most 150 ms slower

def canary_verdict(canary, control):
    if canary.requests < MIN_REQUESTS:
        return "wait"
    canary_errors = canary.errors / canary.requests
    control_errors = max(control.errors / control.requests, 0.001)
    if canary_errors > control_errors * MAX_ERROR_RATIO:
        return "rollback"
    if canary.p99_ms > control.p99_ms + MAX_EXTRA_P99_MS:
        return "rollback"
    if canary.applications < MIN_APPLICATIONS:
        return "wait"
    canary_rate = canary.approved / canary.applications
    control_rate = control.approved / control.applications
    if canary_rate < control_rate * 0.95:
        return "rollback"
    return "promote"

from types import SimpleNamespace as Group
control = Group(requests=160000, errors=80, p99_ms=420, applications=8000, approved=5600)
canary = Group(requests=8400, errors=4, p99_ms=450, applications=420, approved=250)
print(canary_verdict(canary, control))  # approvals 59.5% vs 70%

Run it and it prints rollback. The errors and latency are fine, but only 59.5 percent of canary applications were approved against 70 percent for the control. Change approved to 294 and it prints promote.

The first check returns wait until enough requests arrive, because a handful can look perfect by luck. The max on control_errors sets a floor of 0.1 percent. Without it, a control group with zero errors would turn one canary error into a rollback.

The floor has a cost. While control errors stay under 0.1 percent, the check fires only once canary errors pass 0.15 percent. In the tests, a canary with double the control's tiny error rate still passed.

The approval check is the business check. Release 41 threw few server errors; it simply failed approvals. A check on errors alone would have missed it.

Pause and predict: how many applications does the approval check need?

Approval rates bounce around by chance, so a small canary can fail the check while perfectly healthy. Guess how often a healthy canary fails at 50 applications, before reading the numbers.

A simulation used a 70 percent approval rate and 8,000 control applications. Healthy canaries were rolled back about 30 percent of the time at 50 applications, 14 percent at 200 and 7 percent at 400. At 400, a release 41-style bug was caught in 99.9 percent of runs.

Now the cost. At a made-up 1,500 applications an hour in the daytime, a 5 percent canary sees about 75 an hour, so 400 take over 5 hours. A 25 percent step gets there in about an hour, but exposes five times as many users.

The steps add up too. The loop judges at 5, 25 and 50 percent, and a second simulation with fresh data per step rolled back about 18 percent of healthy releases. Re-judging one step after every 50 more applications raised its false rollbacks from 7 to 10 percent. So decide once per step, at 400 applications.

This is the real trade-off in canary design. Technical checks like errors fill up fast; business checks fill up slowly. The SRE workbook says a canary should be big enough and long enough to represent the full deployment.

Make every database change work with both the old and the new code, using expand and contract. Add the new part first, move the code across, and remove the old part only after the new code has proved itself.

This matters because the database is usually shared. In blue-green, both colours write to it. In canary and rolling, both versions run at the same time. If release 42 drops a column that release 41 reads, switching back to 41 just breaks in a new way.

Kubernetes cannot help here. Its docs say a rollback restores only the Deployment's pod template. Your schema, your data and even a changed replica count stay as they are.

Fowler's blue-green article gives the advice: separate schema changes from application upgrades. First change the schema to support both versions, then deploy the new code. Danilo Sato calls the general pattern parallel change, also known as expand and contract.

It has three phases: expand, migrate and contract. Here is how the team uses it to replace a tenure_weeks column with tenure_days, so 45-day loans can be stored exactly.

1. 1

   Expand (release 42)

   Add a new tenure_days column that may be empty. Release 41 ignores it, so rolling back stays safe.

2. 2

   Write both (release 43)

   New code writes tenure_weeks and tenure_days on every loan, but still reads tenure_weeks. Either version can run against this data.

3. 3

   Backfill

   A background job fills tenure_days for old loans in small batches, so the database never slows down for applicants. Run it again after any rollback to release 41 or 42, because they leave tenure_days empty.

4. 4

   Read the new column (release 44)

   Code now reads tenure_days. It still writes both columns, so release 43 remains a safe rollback target.

5. 5

   Contract (release 46)

   Two quiet weeks later, the team stops writing tenure_weeks and drops it. Only now is there no way back to release 41.

Roll back as soon as a release clearly hurts users, and choose a strategy where that takes seconds. Write down the triggers before the release, so nobody argues about them during an incident.

Speed starts with one habit: never rebuild to go back. Keep every built version, called an artifact, stored and ready. Most of the 50 lost minutes went on rebuilding release 40, not on deciding to roll back.

The team now writes these triggers into each release plan:

• Canary error rate more than 1.5 times the control's, after at least 2,000 requests.
• Canary p99 latency more than 150 ms above the control's.
• Canary approval rate below 95 percent of the control's, after at least 400 applications.
• Any new pod failing its health check for 2 minutes in a row.
• The release engineer or QA lead seeing a bug that touches money, whatever the numbers say.

The first three can run automatically. Sato's canary article describes monitoring business metrics and rolling back automatically. It calls this a cluster immune system and says IMVU pioneered it.

The cost of skipping this is real. In late July 2012, Knight Capital, a US trading firm, incorrectly deployed new code to its order router. The US Securities and Exchange Commission (SEC) described what followed.

On 1 August 2012, in the first 45 minutes after the market opened, the router sent more than 4 million orders. It was trying to fill just 212 customer orders. The firm lost more than 460 million US dollars.

The SEC also said an internal system sent 97 automated emails that flagged an error before the market opened. The firm did not act on them that day. An alert with no agreed trigger and no owner protects nobody.

One more tool helps here. A feature flag is a switch in your own code that turns a feature on or off without a new deployment. The team now ships the new rounding logic behind a flag, so it can be turned off in seconds.

• Agree the triggers and who presses rollback before the release starts.
• Automate the clear triggers; leave judgement calls to named people.
• Practise a rollback on a quiet day, and time it with kubectl rollout status, so the first one is not during an incident.

Choose by two questions: how much harm a bad release can do, and how much traffic you have to judge it. Different services can use different strategies, and often should.

Here is the decision the team wrote down. Start at the top, answer each question, and follow the labelled arrow.

Rolling is a fine choice for the SMS reminder workers. Nobody waits on a screen for them, and a bad version only delays a few reminders.

A canary also needs monitoring split by version, steady traffic, and a way to keep each user on one version. A small team can start with rolling plus health checks, then add canaries where the money flows.

Explain them as two answers to one question: how do you limit the harm of a bad release? Then show the numbers and the database plan that make rollback real.

1. 1

   Start from the risk

   Say what a bad release would break, such as loan approvals, and how many users would feel it per minute.

2. 2

   Separate deploy from release

   Explain that code can sit on servers before users reach it, through a second environment, a small traffic slice or a feature flag.

3. 3

   Pick and justify a strategy

   Canary for busy, risky services; blue-green for low traffic or full pre-switch testing; rolling for everything else. Name the cost of each.

4. 4

   Define the checks

   Compare canary against control on errors, p99 latency and one business number. Say how many requests or applications each check needs.

5. 5

   Plan the database

   Describe expand and contract, and say that destructive changes ship last, in a release of their own.

6. 6

   Show the rollback

   Say how long it takes, who or what triggers it, and that old artifacts stay ready so nothing is rebuilt.