Liveness vs Readiness Probes: Stop Routing to Sick Pods

Users get errors even though every server looks up. A process can be running and listening on its port, yet still unable to answer one request properly. Telling those two states apart is what liveness vs readiness is about.

Picture a common setup. An ambulance booking service runs its dispatch API as 6 copies on Kubernetes, a system that starts, stops and watches containers across many machines. Each copy loads a live map of 140 ambulances and 22 hospitals when it starts, which takes about 40 seconds. The setup is made up, but both failures below are common.

• Night one, a routine deploy. New copies got bookings the moment their process began. For 40 seconds each, they had an empty map and replied that no ambulance was free.
• Night two, a fix gone wrong. The team had added a check that restarted any copy that could not reach the database. The database slowed down for two minutes. All 6 copies restarted together, then kept restarting before their 40-second map load could finish, until the database recovered.

Night one needed a check that holds traffic back until a copy is ready. Night two came from a check that restarted copies for a problem a restart can never fix. Those are the two probes this guide is about, plus a third that protects slow starts.

You need five terms. Each one gets a plain meaning and its place in the dispatch service.

Container and pod. A container is your packaged app running as a process. A pod is the smallest unit Kubernetes runs: one or more containers that live and move together. Each dispatch copy is one pod with one container.

Kubelet. The kubelet is the agent on every machine, called a node, that starts containers and runs their probes. It is the one calling the dispatch health endpoints every few seconds.

Service and EndpointSlice. A Service is one stable address that spreads requests across many pods. An EndpointSlice is the list of pod addresses behind a Service, with a ready flag on each. The booking app always talks to the dispatch Service, never to one pod.

Probe. A probe is how the kubelet asks a container a question. It can send an HTTP GET request, open a TCP connection, run a command inside the container, or call the standard gRPC health service. gRPC is a common way for services to call each other.

For an HTTP probe, any status code from 200 to 399 counts as a pass. Any other code, or no answer within the timeout, counts as a failure.

A liveness probe decides whether to restart a container, and a readiness probe decides whether the pod should receive traffic. Liveness fixes a stuck process by replacing it. Readiness protects users by routing around a pod that cannot serve right now.

Picture a sabzi mandi, a wholesale vegetable market, before dawn. The mukadam, the head of a gang of hamals (porters), calls them one by one to unload trucks. Before he calls a hamal, he is really asking two questions.

First, is he awake at all? A hamal fast asleep on the sacks, who does not stir however loudly he is called, gets sent home and someone fresh takes his place. That is liveness, and the fix is a replacement.

Second, is he free for the next load? He may be wide awake but still carrying the last sack. Nobody sends him home for that; the mukadam simply skips him for now. That is readiness.

Map it back. A dispatch pod loading its map is an awake hamal who is not free yet: alive, not ready. A pod whose main loop has frozen is the hamal who will not wake: it needs a restart.

Where the picture breaks: a hamal sent home is replaced by a different person. A liveness restart reruns the same container image in the same pod on the same node, so a bug that froze it once can freeze it again.

When a readiness probe fails a set number of times in a row (failureThreshold, 3 by default), the kubelet marks the container not ready. The Service then stops sending the pod requests. The container keeps running and keeps being checked.

Once the probe passes again, the pod rejoins and gets traffic. Readiness probes run for the container's whole life, so a pod can leave and rejoin many times.

A pod with a readiness probe starts out not ready, because the kubelet counts readiness as failed until the first check passes. A container with no readiness probe counts as ready as soon as it has started, or once its startup probe passes.

Readiness also paces deploys. A Deployment, the object that rolls out new versions, counts a new pod as available only once it is ready. Without a readiness probe, a rollout can swap warm old pods for cold new ones as fast as containers start. That is exactly night one.

Here is night one redone with a readiness probe. Follow the request from the booking app, and notice that new pod 6 is running but sits outside the EndpointSlice until its map is loaded.

You can watch this happen. While a new pod warms up, kubectl get pods shows it as Running with 0/1 in the READY column. kubectl describe pod lists Unhealthy events such as Readiness probe failed: HTTP probe failed with statuscode: 503.

The probes page says the pod's address is removed from the Service's EndpointSlices. The EndpointSlices page instead describes a serving flag that follows the pod's readiness. In the v1.37.0 source, the entry stays, with its ready and serving flags set to false, and Service proxies skip it.

To see it, run kubectl get endpointslices -o json -l kubernetes.io/service-name=dispatch. Pod 6 appears with "ready": false until its map is loaded.

Readiness has a second job after startup. The documentation suggests it for pods recovering from temporary faults or overloads, and for a container that takes itself out of traffic for maintenance.

When a liveness probe fails failureThreshold times in a row, 3 by default, the kubelet kills the container and restarts it. It is the same pod on the same node, with a fresh process inside. The pod's restartPolicy governs this, and Deployments always use Always.

That is right for a deadlock, where two parts of a program each wait for the other forever. The process looks alive to the operating system but makes no progress. A restart frees it.

It is wrong for problems outside the process. A restart cannot speed up a slow database or mend a broken network. It only throws away a warm pod and makes it reload its map.

Follow the decision the kubelet makes on every liveness check. Only the second question box can lead to a restart, and by default it needs 3 failures in a row.

If a container keeps getting restarted, the kubelet waits longer each time: 10 seconds, then 20, then 40, capped at 5 minutes. kubectl then shows the pod as CrashLoopBackOff. The wait resets once the container runs for 10 minutes without trouble.

Those are the kubelet defaults in v1.37. Admins can lower the cap per node with the kubelet's crashLoopBackOff setting, a beta feature that is on by default since v1.35.

You do not always need a liveness probe. The documentation says that if your process already crashes on its own when it breaks, the restart policy handles it. Liveness is for apps that hang instead of dying.

A startup probe checks only whether the app has finished starting. Until it passes once, the kubelet does not run the liveness or readiness probes at all. You need one when startup takes longer than liveness would allow.

Without one, the 40-second map load clashes with liveness. The dispatch health endpoint answers 503 until its map is loaded. With default liveness settings, the kubelet would kill the container 20 to 30 seconds in, on every start.

In the mandi, this is the new hamal who gets a few minutes to settle in before the mukadam starts calling him.

The startup probe gives a fixed budget: failureThreshold times periodSeconds, the gap between checks. Checking every 5 seconds with 24 allowed failures gives about 120 seconds. If it never passes in that time, the container is killed and restarted.

Why not set a long initialDelaySeconds on liveness instead? That delay is paid on every start, even a fast one, and liveness is blind during it. A startup probe ends at its first success, so liveness takes over as soon as the app is up.

Read the timeline from top to bottom. Only one probe runs at the start. The other two take over once startup passes.

A liveness endpoint should check only whether this process is making progress. A readiness endpoint should check whether this pod can do useful work right now. Keep both cheap, because the kubelet calls them every few seconds on every pod.

Here are the dispatch service's health endpoints in Flask 3.1, a small Python web framework. The dispatch loop is a background thread that matches bookings to ambulances. It writes the time of each pass into state, a shared dictionary.

import threading
import time

from flask import Flask

app = Flask(__name__)
state = {"last_loop": None}  # the dispatch loop writes here on every pass


def load_map():
    time.sleep(40)  # stand-in for loading 140 ambulances and 22 hospitals


def match_bookings():
    pass  # stand-in for the real matching work


def dispatch_loop():
    load_map()
    while True:
        state["last_loop"] = time.monotonic()
        match_bookings()
        time.sleep(1)


def seconds_since_loop():
    last = state["last_loop"]
    return None if last is None else time.monotonic() - last


@app.get("/livez")
def livez():
    age = seconds_since_loop()
    if age is None:
        return "starting", 503
    if age > 30:
        return "dispatch loop stuck", 500
    return "ok", 200


@app.get("/readyz")
def readyz():
    age = seconds_since_loop()
    if age is None:
        return "loading ambulance map", 503
    if age > 10:
        return "dispatch loop behind", 503
    return "ready", 200


if __name__ == "__main__":
    threading.Thread(target=dispatch_loop, daemon=True).start()
    app.run(host="0.0.0.0", port=8080)

What just happened? Both endpoints read one signal: how long ago the loop last ran. time.monotonic is a clock that never jumps backwards when the system time changes. Readiness gives up after 10 seconds behind, liveness only after 30 seconds stuck.

Look at what /livez does not touch. It never calls the database or another service. It only asks whether this process is moving, the one problem a restart can fix.

You can check it on your laptop. After pip install flask==3.1.3, run python dispatch_health.py, then curl -i localhost:8080/readyz. For the first 40 seconds you get HTTP/1.1 503 SERVICE UNAVAILABLE and loading ambulance map.

After that, both endpoints answer 200 OK. When we froze the loop in a test, /readyz turned 503 after 10.0 seconds and /livez turned 500 after 30.2 seconds.

The sample runs on Flask's built-in development server, in one process, to stay short. A production server that forks several worker processes would need the dispatch thread started inside each worker.

• Check in liveness: the main loop or worker threads are still moving.
• Check in readiness: data the pod loads at start, such as maps, caches or models, is in memory.
• Check with care in readiness: a dependency that every pod shares, such as the database.
• Never check in liveness: databases, queues or other services.
• Never do in either: slow work, such as a full query or a call that can take seconds.

Should readiness check the shared database?

The documentation allows it: readiness may also check that each required back-end service is available. The catch is that a database every pod shares fails for every pod at the same moment.

Then every pod goes not ready together, and the Service has no ready pods left to send requests to. Callers get no answer from the dispatch pods at all, not even a helpful error.

The team kept the database out of readiness. When the database is down, each pod stays ready and answers bookings at once with a clear 503 and a fallback message. An alert on that error rate pages the on-call engineer.

A bad liveness probe takes down a service when it fails for a reason every pod shares. All pods restart together, capacity drops to zero, and the restarts add load of their own.

The Kubernetes documentation warns that incorrect liveness probes can lead to cascading failures, where one failure triggers the next. Containers restart under high load, client requests fail, and the remaining pods take on more work.

This is not only a Kubernetes problem. The Google SRE book describes schedulers killing overloaded tasks that fail health checks, which overloads the tasks left behind. It treats process health and service health as two distinct checks: one for the scheduler, one for the load balancer.

Trace night two in the diagram. Start at the slow database. Every arrow after it comes from the liveness probe, not from the database.

Every restart threw away a half-loaded map, and between later attempts kubectl showed the pods in CrashLoopBackOff. Had the database check stayed out of liveness, the pods would have stayed warm. Bookings would have failed fast for two minutes, then recovered with no map reload at all.

Kubernetes acts after failureThreshold failed checks in a row, spaced periodSeconds apart. A problem can start anywhere between two checks. So the wait is between failureThreshold minus 1 and failureThreshold periods, plus up to one timeout.

These are the timing fields and their defaults in Kubernetes v1.37, the newest minor release as of 19 October 2026:

• periodSeconds: how often the probe runs. Default 10 seconds, minimum 1.
• timeoutSeconds: how long one check may take before it counts as a failure. Default 1 second, minimum 1.
• failureThreshold: how many failures in a row count as failed. Default 3, minimum 1.
• successThreshold: how many passes in a row count as passing again. Default 1, and it must be 1 for liveness and startup probes.
• initialDelaySeconds: how long to wait after the container starts before the first check. Default 0.

Here is the probe section from the dispatch container spec. The values were chosen for this app, not copied from defaults.

startupProbe:
  httpGet: {path: /livez, port: 8080}
  periodSeconds: 5
  failureThreshold: 24    # 24 x 5 s: up to about 120 s to start
readinessProbe:
  httpGet: {path: /readyz, port: 8080}
  periodSeconds: 5
  failureThreshold: 2     # 2 misses in a row: out of traffic
livenessProbe:
  httpGet: {path: /livez, port: 8080}
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 6     # 6 misses in a row: restart

Pause and predict: the dispatch loop freezes at exactly 2:00:00 a.m. When does the pod leave traffic, and when is it restarted?

Readiness goes first. /readyz starts failing 10 seconds after the freeze, and two failures 5 seconds apart must follow. The pod is marked not ready 15 to 20 seconds after the freeze.

Liveness is slower on purpose. /livez fails only after 30 seconds stuck, then needs 6 failures 10 seconds apart. The kubelet decides to restart 80 to 90 seconds after the freeze. The old container then gets a grace period, 30 seconds by default, to exit before it is killed.

That gap is the pattern the documentation describes: a pod is seen as not ready for a while before it is hard killed. A pod that is merely busy gets time to recover, out of traffic, with its map intact.

Details the docs do not spell out

Two behaviours come from reading the kubelet source at v1.37.0, not the documentation, as of 19 October 2026. Pin your version before relying on them.

• Liveness and startup probes run on their own ticker, which the kubelet starts when it sets up the pod, before the container runs. So their first check lands up to one full period after the container starts, or after the startup probe passes, not at second zero.
• When a pod is being deleted, the kubelet stops its liveness and startup probes and records them as passing. The readiness probe keeps running during shutdown.

Readiness is different, and the docs do note it. While a container is not ready, the kubelet may also run its readiness probe between ticks, to make the pod ready sooner. In the source, this happens right after the container starts or its startup probe passes.

When a pod is deleted, two things start at the same time. The kubelet begins shutting the pod down, and its endpoint is marked not ready so load balancers stop using it for new traffic.

The kubelet first runs any preStop hook, a command you can set to run before shutdown. Then it sends SIGTERM, a polite request to stop, to the main process in each container. By default the whole shutdown gets 30 seconds before anything left is killed.

Nothing orders the routing change and SIGTERM, so a request routed a moment earlier can still arrive after SIGTERM. The pod should finish bookings already in progress before it exits.

One way to buy time is a preStop hook using the built-in sleep action. The kubelet then pauses for a few seconds before SIGTERM, while routing catches up. Keep the sleep well inside the grace period.

Check that your app really exits on SIGTERM: run as process 1, the first process in a container, our Flask sample ignored it and kept answering. Process 1 ignores signals it has no handler for, and Python sets no SIGTERM handler. Adding signal.signal(signal.SIGTERM, lambda signum, frame: sys.exit(0)) to the __main__ block, after importing signal and sys, made it exit at once; a real service should finish bookings in progress first.

Make each check fail on purpose in a test cluster and watch what Kubernetes does. A probe you have never seen fail is a guess.

1. 1

   Freeze the main loop

   Add a test-only switch that pauses the dispatch loop. kubectl get pods should show 0/1 ready within about 20 seconds. About 80 to 90 seconds after the freeze, kubectl describe pod shows Container dispatch failed liveness probe, will be restarted. RESTARTS goes up by 1 once the old process exits, which can take up to the 30-second grace period.

2. 2

   Slow down startup

   Make the map load take 90 seconds. The pod should become ready roughly 90 to 100 seconds after it starts, with RESTARTS still 0, inside the 120-second budget.

3. 3

   Cut off the database

   Block the test pods from the database. Pods should stay Running and ready with zero restarts, answer bookings with a fast 503, and fire the alert. kubectl describe pod should show no liveness failures at all.

4. 4

   Deploy under load

   Send a steady stream of test bookings and roll out a new version. No booking should fail while old pods drain and new ones warm up.

The database drill is the one that would have caught night two. A shared dependency failing should never restart a single pod.

Explain it as two questions with two different actions, then show how each one fails in a real system.

1. 1

   Define both in one line each

   Liveness decides whether to restart a container. Readiness decides whether the pod gets traffic. Mention that a startup probe holds both back while a slow app starts.

2. 2

   Use a warm-up example

   Describe a service that loads data at start, like the dispatch map. Show readiness keeping it out of traffic until the load is done.

3. 3

   Name what each probe checks

   Liveness checks only the process itself. Readiness checks warm-up and whether the pod can do useful work now.

4. 4

   Show the cascading failure

   Explain how a liveness probe on a shared database can restart every pod at once, and why a restart cannot fix a slow database.

5. 5

   Talk about timings

   Readiness should react fast and liveness slowly. Give the time to act as roughly failureThreshold times periodSeconds.

6. 6

   Cover shutdown

   Say that a deleted pod is marked not ready, gets SIGTERM, and has a grace period, 30 seconds by default, to finish in-flight requests.