Chaos Engineering and Fault Injection: Break It on Purpose

It usually fails on the day you need it. Code that handles failure is still code, and code that has never run tends to break in ways nobody expected. Chaos engineering is the practice of running that code on purpose, safely, before a real outage does it for you.

Picture a common setup. A scooter rental app serves riders in a few hill towns, such as Dehradun and Mussoorie. Riders find a free scooter, book it, open its lock from the app and pay by the hour. The setup is made up, but the problem is real.

The app has a few backend services. A service is one program with one job that other programs call over the network. The booking service takes the order. The location service finds free scooters near the rider. The lock service talks to the lock box on each scooter.

One monsoon evening, the location service became slow. It did not crash. It took 25 to 30 seconds to answer instead of a quarter of a second.

The team had written a fallback a year before. A fallback is a backup path the code takes when a call fails. This one used the last saved list of nearby scooters from a cache, a fast store of recent data.

The fallback never ran. It was written to trigger on errors, and a slow answer is not an error. The booking service has a fixed number of workers. Each worker is a thread, a line of work inside the program that handles one request at a time.

Every worker got stuck waiting up to 30 seconds for the location service. New requests had no free worker, and bookings stopped for 52 minutes. About 340 riders stood in the rain next to scooters they could not book.

The incident review found one plain fact. The team had a backup plan, but nobody had ever seen it work. The rest of this guide follows how they fixed that.

Chaos engineering is the practice of running careful experiments on a system to build confidence that it copes with bad conditions in production. You break one thing on purpose, measure what users experience, and learn from the result.

Production means the live system that real users touch, as opposed to a test copy. The published Principles of Chaos Engineering define the practice the same way: experiments that build confidence a system can withstand turbulent conditions in production.

The word chaos is a little misleading. Nothing about a good experiment is random or wild. It is closer to a science test, with a clear question and a clear way to stop.

Think of a family in Dehradun before the monsoon. They own an inverter, a battery box that should keep the fridge and lights running in a power cut. On a calm afternoon, they switch off the mains on purpose and watch for ten minutes.

Map it part by part. The mains switch is the fault you inject. The fridge and lights are the steady state you measure, because that is what the family cares about. The calm afternoon, with someone standing at the switch, is the small and safe blast radius.

Where the picture breaks: a house has one inverter and nothing else depending on it. A real system has many services that slow each other down in ways no single switch shows. Real users are also on it while you test.

Netflix engineers described the practice in a 2016 IEEE Software article titled Chaos Engineering. Its abstract says many large tech organisations use experimentation to verify the reliability of their distributed systems.

A distributed system is one made of many programs on many machines, talking over a network. The rental app is a small one. Such systems fail in ways that are hard to predict by reading code, which is why experiments help.

Fault injection means deliberately adding a failure to one part of a system, such as a crash, a delay or an error reply. It is the main tool of chaos engineering: the experiment is the plan, and fault injection carries it out.

The Principles suggest events that mirror real life. Their examples are hardware failures like servers dying, software failures like malformed responses, and non-failure events like a spike in traffic. Pick the events most likely to happen, or most costly if they do.

Here are the faults the rental team listed, from simplest to hardest:

• Kill a server. Stop one copy of the booking service and check that the other copies take its traffic.
• Add latency. Latency is how long one request takes. Make the location service answer 2 seconds late, the fault from the monsoon evening.
• Return errors. Make payment calls fail for a small group of test riders, and check that each rider sees a clear message.
• Cut the network. Block traffic between the booking service and the cache, so the fallback has nothing to read.
• Lose a whole zone. A zone is one data centre area at a cloud provider. Turn off everything the app runs in one zone and see if the rest copes.

Notice the second item. A slow service is often worse than a dead one. When a service has crashed but its machine is still up, connections are refused at once, so callers move on. A slow one holds each caller's worker until it answers or the caller gives up, and the engineer section puts numbers on that.

There are three common places to put a fault. You can add it inside your own code behind a switch, in a proxy (a small program that relays traffic between services), or through a cloud provider's fault tool. The rental team starts with a switch in the location service.

The diagram shows the booking path and where the fault goes. Since the outage, the booking service has a timeout: it gives the location service 0.8 seconds to answer, then falls back to the cache.

Nobody has seen that timeout and fallback work under a real slowdown, so they are what the experiment really tests. The fault switch itself only changes the location service.

You run it like a science experiment: define normal, predict what will happen, inject one fault, and compare. The Principles give four steps, and teams add planning before them and fixing after.

First, define a steady state. Second, predict it will continue in both a control group and an experimental group. Third, introduce real-world events. Fourth, try to disprove the prediction by looking for a difference between the groups.

A steady state is a measurable output that shows the system is behaving normally. For the rental app, it is the share of booking attempts that succeed, normally about 98.5 percent, and how fast bookings finish.

The Principles advise measuring a system's output, not its internal details. Their examples are throughput (work done per second), error rates and latency percentiles. A latency percentile such as p99 is the time within which 99 of every 100 requests finish.

CPU use, by contrast, tells you how busy a part is, not whether riders can book. In the inverter test, the family watches the fridge, not the battery voltage.

1. 1

   Pick one question

   The team picks the monsoon fault. If the location service slows down, will riders still book using the saved list?

2. 2

   Define the steady state

   At least 98.5 percent of booking attempts succeed, and 99 percent finish within 3 seconds. Both numbers already sit on the team's dashboard.

3. 3

   Write the hypothesis

   A hypothesis is a guess you can prove wrong. The team writes: with every location call from the experiment group delayed by 2 seconds, the steady state will not change.

4. 4

   Split into two groups

   A random 5 percent of riders form the experiment group and get the fault. Another random 5 percent form the control group and get none. Comparing the two separates the fault's effect from noise like rain or a festival rush.

5. 5

   Inject the fault and watch

   The team turns the switch on for up to 30 minutes. One person watches the stop conditions and holds the abort switch.

6. 6

   Try to disprove the guess

   If the experiment group's numbers drop while the control group's hold, the hypothesis was wrong. That is a finding, not a failure.

7. 7

   Fix and repeat

   Write down what broke, fix it, and run the same experiment again. A passing experiment then becomes a regular, automated check.

This picture shows the same loop as a cycle. The comparison box is the decision point.

Teams often write the plan as a short file, so others can review it like code. Open-source runners such as Chaos Toolkit read that file, check the steady state, inject the fault and undo it. The engineer section runs one on a laptop.

You keep it safe by limiting the blast radius, setting automatic stop conditions, and having a person ready to abort. All three are designed in before the experiment runs.

Blast radius means how many users or systems an experiment can affect if it goes wrong. The Principles call it the chaos engineer's responsibility to keep the fallout small and contained, since some short-term harm can happen.

• Start tiny. The first run touches the 5 percent experiment group only, not every rider in every town.
• Keep it short. Thirty minutes at most is enough to see a large effect, and short enough to limit damage.
• Set stop conditions. If experiment-group success falls below 90 percent over at least 20 bookings, the script turns the switch off by itself.
• Name an abort owner. One person does nothing during the test except watch and decide, and can stop it at any time.
• Choose the time. Run it on a weekday afternoon, when the team is awake and at desks, never during a festival rush.
• Tell people first. Support staff and the on-call engineer know the plan, so nobody mistakes the test for a real outage.

The minimum of 20 bookings matters more than it looks. A stop rule checked on a handful of requests fires on ordinary noise, and the engineer section shows how often.

Cloud fault tools build the same guardrails in. In AWS Fault Injection Service, a stop condition is a CloudWatch alarm, a rule that fires when a metric crosses a threshold you set. If the alarm fires during an experiment, the service stops it.

As of 19 October 2026, the AWS documentation adds that a stopped experiment cannot be resumed. If an alarm fires, you investigate and start a fresh run, rather than pushing on.

This diagram shows the rental app's safety checks while the fault is live. Look at the decision box: it re-checks the numbers after every experiment-group booking.

Some tools offer a dry run. Chaos Monkey has a setting called leashed: when it is true, terminations are only simulated and logged. As of 19 October 2026, its source code sets leashed to true and enabled to false by default.

It disproved the hypothesis: some riders in the experiment group could not book. That is exactly what a good experiment can do.

The stop condition ended the run after about 7 minutes, when 4 of the first 20 experiment-group bookings had failed (made up). The control group stayed at its normal level, so the fault caused the drop, not the weather.

The record told the story. The fallback read the saved scooter list from the cache, and each saved list expired 60 seconds after it was written.

Busy pickup points get bookings every few seconds, so their saved list was always fresh. Quiet pickup points in Mussoorie can go many minutes between bookings. There, the saved list had expired, the fallback found nothing, and it returned an error.

The team changed the fallback to keep a stale list for up to 10 minutes and mark it as possibly old. The app now shows riders a short note that availability may be a little out of date. They also added an alert that fires when fallback use rises.

The second run, a week later, held the steady state: both groups stayed near 98.5 percent (made up). The team then grew the experiment group to 10 percent, and later scheduled the test every Tuesday.

Eventually, yes, but not first. The Principles strongly prefer experimenting on production traffic. Our advice is more cautious: earn production by first proving the fault, the stop conditions and the abort all work in staging.

Staging is a copy of production used for testing. It is never quite the same: it has less traffic, smaller machines and different data. The Principles argue that only sampling real traffic reliably captures the paths requests really take.

The rental app shows why. The bug lived at quiet pickup points, a pattern that depends on how real riders spread across real towns. A staging copy with test traffic would likely never have shown it.

So the team's path was staging until the fault, the stop conditions and the abort all behaved. Then production with a 5 percent group and a short run. They grew the group only after clean runs.

Teams use tools that kill machines at random, tools that inject precise faults, and planned team exercises called game days. A game day is an easy first step, because it needs no new software.

Chaos Monkey is the best-known tool. Netflix built it and released it as open source. Its documentation says it randomly terminates instances in production, so engineers build services that survive losing one.

An instance is one running copy of a service. It runs on a virtual machine, a computer simulated in software, or in a container, a packaged app isolated on a shared machine. As of 19 October 2026, Chaos Monkey needs Spinnaker, Netflix's open-source deployment platform, and a MySQL-compatible database, version 5.6 or later. MySQL is an open-source database.

It does not run as an always-on service. A cron job, the Unix way to run a command on a timetable, starts it once each weekday to plan that day's terminations.

AWS Fault Injection Service is a managed option inside AWS. As of 19 October 2026, its action list includes stopping virtual machines, CPU stress and added network delay for containers, and injected errors on some AWS API calls. Those errors cover only the EC2 and Kinesis services, AWS's virtual machines and data streams.

A service mesh is a layer of small proxies that carries traffic between services. Istio, an open-source mesh, can add a fixed delay or an error reply to matching requests through configuration alone, with no code change.

Chaos Toolkit is an open-source runner for experiment files. It checks the steady state, runs the fault, checks again and rolls back, which means it undoes the fault. Version 1.21.4 was the latest on PyPI, Python's package index, as of 19 October 2026.

A game day is a planned session where the team breaks something on purpose and practises the response together. The rental team runs one each quarter, timing how long they take to notice, find the cause and recover.

Game days test people as much as software. At the team's first one, the alert reached the right phone, but the runbook pointed to a dashboard that no longer existed. A runbook is the written checklist for handling a known problem.

Normal tests check known behaviour on a test copy and pass or fail. Chaos engineering explores unknown behaviour under real failure, often in production, and produces learning. The two work together rather than replace each other.

The table adds load testing, which checks how a system copes with heavy traffic.

The most common mistakes are starting too big, skipping the hypothesis, and never fixing what the experiment finds. Each one turns a learning tool into a source of outages or busywork.

• Starting with a zone outage. Begin with one instance or one delayed call, then grow.
• No steady state. Without a number for normal, you cannot tell whether the fault did harm.
• No control group. Without one, a rainy evening's drop in bookings looks like your fault's effect.
• Ignoring slowness. Teams test crashes but forget delays, which caused the monsoon outage.
• Findings with no owner. Each finding needs a named person and a date, or it is forgotten.
• Running once and stopping. Code changes every week, so a fallback that worked in March can break by May.

The last point is why the Principles advise automating experiments to run continuously. Manual experiments take effort, so they happen rarely. Automated ones catch the bug that a new deploy quietly added.

Because each waiting call holds a worker for as long as it waits. Little's law, a basic queueing result, says the average number of busy workers equals requests per second times seconds each request holds a worker.

Say the booking service gets 10 requests a second that call the location service, and has 200 workers across its copies (made up). At the normal 0.25 seconds, about 2.5 workers are busy on average. At 30 seconds, it would need 300.

With only 200 workers, the pool fills in about 20 seconds, which is 200 divided by 10. Every request after that must wait in line or be turned away, including requests that never touch the location service. That is how one slow service stopped all bookings.

Read the last column first. Below the pool size, nothing waits. Above it, requests pile up fast. The 0.8 second timeout caps how long a slow location call can hold a worker. However slow location gets, the damage stays at or below about 8 busy workers on average, as long as the timeout covers the whole call, not each read.

Timeouts must shrink down the call chain

A request often passes through several services, each with its own timeout. If an inner service waits longer than its caller will, the caller gives up first, and the inner fallback never helps.

Istio's own fault injection task shows this, in its docs for version 1.31 as of 19 October 2026. It delays calls from the reviews service to the ratings service by 7 seconds, for one test user, in the Bookinfo demo app. Reviews has a hard-coded 10 second timeout, so it would wait.

But productpage, the service in front, gives reviews 3 seconds plus one retry, about 6 seconds in all. For the test user, the page loads after about 6 seconds, but its reviews section shows an error. The docs call this a bug found by the test. A later reviews version cuts its timeout to 2.5 seconds.

Fewer than you fear when the harm is large, and far more than you expect when it is small. A small blast radius means few requests, so small drops hide inside normal noise.

In the rental app, each 5 percent group sees about 3 booking attempts a minute on a weekday afternoon (made up). The question is how many attempts each group needs before a drop from 98.5 percent stands out.

The script below uses the standard formula for comparing two success rates. It assumes 95 percent confidence, so a healthy run is wrongly flagged about 1 time in 20. It also assumes 80 percent power, so a real drop of that size is caught about 8 times in 10.

from math import ceil
from statistics import NormalDist

def bookings_per_group(normal, harmed, confidence=0.95, power=0.80):
    z = NormalDist().inv_cdf
    za, zb = z(1 - (1 - confidence) / 2), z(power)
    spread = normal * (1 - normal) + harmed * (1 - harmed)
    return ceil((za + zb) ** 2 * spread / (normal - harmed) ** 2)

for harmed in (0.97, 0.95, 0.90, 0.80):
    print(f"98.5% -> {harmed:.1%}: {bookings_per_group(0.985, harmed)} bookings per group")

On Python 3.13 it prints 1531, 400, 114 and 41 bookings per group, for drops to 97, 95, 90 and 80 percent. A simulation of 20,000 runs at the 41-booking size caught the 80 percent drop 88 percent of the time, so the formula is slightly cautious here.

At 3 attempts a minute, catching a drop to 80 percent takes about 14 minutes. Catching a drop to 97 percent takes about 8.5 hours. Size the experiment for the harm you care about, or accept that small harms will slip through.

Stop conditions need a minimum count

Stop rules face the same noise. Suppose a script checks each minute and stops when that minute's success is under 97 percent. With 3 attempts a minute, a single ordinary failure is enough to trip it.

At a healthy 98.5 percent, that rule fires in about 4.4 percent of minutes. Over a 30-minute run it ends about 74 percent of perfectly healthy experiments, and a team soon learns to ignore it.

The rental team's rule waits for at least 20 bookings, then stops if overall experiment-group success is below 90 percent. In 200,000 simulated healthy 30-minute runs it fired about 1 percent of the time. It caught a drop to 80 percent in 99.9 percent of runs, usually as soon as the 20-booking minimum was reached.

Install Chaos Toolkit, start a tiny service with a fault switch, and let the toolkit check the steady state before and after the fault. Each run takes about 7 seconds and needs Python 3.12 or later on Linux or macOS.

The file app.py below plays both services from the story on port 8000. The /location path answers with two scooters, but sleeps 2 seconds while a file named fault.on exists. That file is the fault switch.

The /book path calls /location with a 0.8 second timeout. If the call times out, it uses the saved list, but only if the list is younger than CACHE_TTL seconds. Otherwise it returns status 503, meaning the service is unavailable.

# app.py: a tiny booking service with a fault switch (Python 3.12 or later)
import json, os, time, urllib.request
from http.server import ThreadingHTTPServer, BaseHTTPRequestHandler

CACHE_TTL = float(os.environ.get("CACHE_TTL", "5"))  # seconds a saved list stays usable
saved = {}                                            # pickup point -> (time saved, scooters)

class Handler(BaseHTTPRequestHandler):
    def reply(self, status, body):
        data = json.dumps(body).encode()
        self.send_response(status)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(data)))
        self.end_headers()
        self.wfile.write(data)

    def do_GET(self):
        if self.path == "/location":                  # the location service
            if os.path.exists("fault.on"):
                time.sleep(2)                         # the injected slowness
            return self.reply(200, {"scooters": ["S1", "S2"]})
        if self.path == "/book":                      # the booking service
            try:
                with urllib.request.urlopen("http://127.0.0.1:8000/location", timeout=0.8) as r:
                    scooters = json.load(r)["scooters"]
                saved["point"] = (time.time(), scooters)
                source = "live"
            except OSError:                           # timed out: use the fallback
                entry = saved.get("point")
                if entry is None or time.time() - entry[0] > CACHE_TTL:
                    return self.reply(503, {"error": "no scooters found"})
                scooters, source = entry[1], "fallback"
            return self.reply(200, {"booked": scooters[0], "source": source})
        self.reply(404, {"error": "not found"})

    def log_message(self, *args):
        pass                                          # keep the terminal quiet

ThreadingHTTPServer(("127.0.0.1", 8000), Handler).serve_forever()

The experiment file is written in YAML, a plain-text settings format. It tells Chaos Toolkit what normal looks like: a request to /book returns status 200. It also says which fault to add (create fault.on), how long to wait (6 seconds) and how to undo it (delete fault.on).

title: Bookings survive a slow location service
description: Delay every location call by 2 s and check that booking still works.
steady-state-hypothesis:
  title: Booking works
  probes:
    - type: probe
      name: book-a-scooter
      tolerance: 200
      provider:
        type: http
        url: http://127.0.0.1:8000/book
        timeout: 3
method:
  - type: action
    name: slow-down-location
    provider:
      type: process
      path: touch
      arguments: fault.on
    pauses:
      after: 6
rollbacks:
  - type: action
    name: remove-fault
    provider:
      type: process
      path: rm
      arguments: -f fault.on

Save both files in one folder, then run these commands. The comments show what we saw on Chaos Toolkit 1.21.4 with Python 3.13 in October 2026.

python3 -m venv venv
venv/bin/pip install chaostoolkit==1.21.4
CACHE_TTL=5 python3 app.py &          # start the app; saved lists go stale after 5 s
sleep 1
venv/bin/chaos run experiment.yaml    # near the end: status: deviated
echo $?                               # prints 1
kill $!                               # stop the app
CACHE_TTL=600 python3 app.py &        # the fix: keep saved lists for 10 minutes
sleep 1
venv/bin/chaos run experiment.yaml    # near the end: status: completed
echo $?                               # prints 0
kill $!

Chaos Toolkit then ran the rollback and deleted fault.on, even though the run deviated. The exit code, the number a command returns when it ends, is 1 for deviated and 0 for completed. So a build pipeline can fail on a weakness.

If the app is still running, a BrokenPipeError may print about a second after the deviated run. That is the location handler finishing its 2 second sleep after the caller had already left.

That message shows a real cost of in-code faults: the delayed handler holds a location worker for the full 2 seconds. Proxy and mesh faults add the delay outside your application's process instead.

With its defaults, at night: between 9:30 pm and 3:30 am IST while the US is on daylight time. Its start_hour of 9 and end_hour of 15 are read in the time_zone setting, which defaults to America/Los_Angeles.

US daylight time ends on 1 November 2026. From then, the same window becomes 10:30 pm to 4:30 am IST. Remember that Chaos Monkey kills nothing until you set enabled to true, leashed to false and schedule_enabled to true. Even then, it only picks from groups of copies that Spinnaker marks as enabled.

The fix is one setting: time_zone set to Asia/Kolkata, so 9 to 15 means Indian working hours. We read these defaults in the project's source file config/monkey.go and converted the hours with Python, as of 19 October 2026.

You can check the conversion yourself. This Python 3.9 or later snippet prints the start of the default window in IST:

from datetime import datetime
from zoneinfo import ZoneInfo

start = datetime(2026, 10, 19, 9, tzinfo=ZoneInfo("America/Los_Angeles"))
print(start.astimezone(ZoneInfo("Asia/Kolkata")))  # 2026-10-19 21:30:00+05:30

One more detail matters on servers in India. Chaos Monkey's deploy guide assumes the cron host's clock runs on UTC (Coordinated Universal Time, the world reference clock; IST is UTC plus 5.5 hours). As of 19 October 2026, its source also writes each planned termination into cron in UTC hours. On a cron host set to IST, each of those lines would fire 5.5 hours early, so keep that host on UTC.

How often does it strike? Each weekday, for every instance group (a set of copies it treats as one pool), Chaos Monkey flips a weighted coin. The chance is 1 divided by the app's mean time between kills, counted in work days.

With a mean of 2, a group loses an instance on about half of weekdays, around 2.5 a week. On a yes, it picks a random minute between start_hour and end_hour.

Explain it as a safe, measured way to prove your design's failure handling works. Tie each experiment to a part of your design, such as a timeout, a replica (a spare copy) or a fallback.

1. 1

   Name the risky dependency

   Point to one call in your design that could fail or slow down, like the location service in a rental app.

2. 2

   Show the defence

   Describe the timeout, retry limit, circuit breaker (a switch that stops calling a failing service for a while) or fallback that protects users. Check that inner timeouts are shorter than outer ones.

3. 3

   State the steady state

   Pick a user-facing number, such as booking success rate or p99 latency, that shows the system is healthy.

4. 4

   Describe the experiment

   Inject a delay or error for a small experiment group, for a short time, with a control group for comparison.

5. 5

   Explain the guardrails

   Mention stop conditions with a minimum request count, an abort owner, working hours and a tested off switch. This is where many candidates fall short.

6. 6

   Close with the follow-up

   Say what you would do with a failed hypothesis: fix it, add an alert, rerun, then automate the test on a schedule.

If the interviewer asks why not just write more unit tests, give the monsoon story in two sentences. The fallback code was correct, but it never ran, because the failure was slowness and not an error.