The Knight Capital Glitch Explained: Dead Code and a Bad Deploy

Old code that nobody deleted can start running again when one server misses a deploy and a reused setting points at it. That is how Knight Capital lost over $460 million in about 45 minutes, in what is now called the Knight Capital glitch.

Picture a common setup. A small trading team runs an order router, the program that sends its buy and sell orders to the stock exchange, on six servers. The setup is made up, but the problem is real.

The team's next release adds a new way to split large orders. To save time, a developer reuses an old setting called use_sweep, which belonged to a feature the team retired years ago. Its code was never deleted.

Someone copies each release to the six servers one at a time, and nobody checks afterwards. Knight Capital made each of these choices in 2012. This guide reads the SEC's record of what followed, then shows what the team changes before it ships.

You need six words. Each gets a plain meaning and an example from the team's router.

Order router. An order router is a program that takes an order and decides how and where to send it to the market. Example: the team's router sends a sell order to whichever exchange shows the best price.

Parent and child orders. A parent order is the full amount someone wants to trade. Child orders are the smaller pieces the router actually sends. When a child order trades, it is filled.

For example, a parent order to buy 10,000 shares might go out as ten child orders of 1,000 each. The router should stop once 10,000 shares have been filled.

Deployment. A deployment is the act of putting new code onto the servers that run it. Example: the team's next release is a deployment to six servers.

Feature flag. A feature flag is an on/off setting that switches a piece of code on without a new deployment. Example: use_sweep set to yes turns on whatever code reads that setting. Knight's flag was a marker sent with each order instead, but the risk is the same: each server's code decides what it means.

Dead code. Dead code is code that nobody uses any more but that still sits in the program. If it can still be called, it is not really dead. It is asleep.

Kill switch. A kill switch is a single control that stops a system from sending anything, at once. Example: one button that cuts all of the team's connections to the exchange.

The Knight Capital glitch was a software failure at a large US trading firm on 1 August 2012. Its order router sent millions of unwanted orders into the stock market in about 45 minutes.

Knight Capital Americas was a broker-dealer with its main operations in Jersey City, New Jersey. A broker-dealer is a firm that trades shares for its customers and for itself. The SEC, the US securities regulator, says Knight generally handled about ten percent of all trading in US listed shares in 2011 and 2012.

The router at the centre of the story was called SMARS. The SEC describes it as an automated, high-speed, algorithmic router. It took parent orders from the rest of Knight's platform and sent child orders to exchanges and other trading venues.

Follow the arrows from the brokers. Notice where the only tool watching positions sits.

In all, the SEC counts 212 incoming parent orders processed by the faulty Power Peg code. It ran first on the eighth server; after the rollback, it ran on the other seven too. For those orders alone, SMARS sent millions of child orders. The SEC counted over 4 million executions, meaning trades that actually happened, in 154 stocks for more than 397 million shares.

When the orders stopped, Knight held a net long position of about $3.5 billion in 80 stocks. Net long means it owned more of those shares than it had sold, here shares it never meant to buy. It also held a net short position of about $3.15 billion in 74 stocks, meaning it had sold more of those shares than it owned.

The SEC says Knight lost over $460 million from these positions. Knight's own filing on 6 August 2012 reported a pre-tax loss of about $440 million. The two figures come from different documents, so this guide uses the SEC's.

The orders also moved prices for everyone else. For 75 stocks, Knight's executions were more than 20 percent of the trading volume and helped move prices by more than five percent.

The failure was built over nine years and released in one morning. The trigger was a new New York Stock Exchange program, the Retail Liquidity Program (RLP), due to start on 1 August 2012. Knight changed SMARS so its customers could take part.

1. 1

   2003: Power Peg is switched off

   Knight stops using an old SMARS feature called Power Peg. The code stays on the servers, present and callable.

2. 2

   2005: the fill counter moves

   Knight moves Power Peg's counter of filled shares to an earlier point in the SMARS code. Knight does not retest Power Peg afterwards.

3. 3

   From 27 July 2012: the new code goes out

   Knight deploys the new retail program code in stages, a few servers a day. A technician does not copy it to one of the eight servers.

4. 4

   1 August, from about 8:01 a.m.: warning emails

   An internal system emails staff about an error it calls "Power Peg disabled". It sends 97 such emails before the 9:30 a.m. open. Knight had not designed them as alerts, and staff generally did not read them.

5. 5

   9:30 a.m.: the market opens

   Seven servers handle the new orders correctly. On the eighth, flagged parent orders reach Power Peg, which sends child orders nonstop for each one.

6. 6

   The next minutes: a live hunt

   Staff search for the cause while Knight stays connected to the market. They remove the new code from the seven good servers, which makes things worse.

7. 7

   About 45 minutes in total

   In all, 212 parent orders reach Power Peg and the SEC counts over 4 million executions in about 45 minutes. Knight is left holding positions it never wanted and loses over $460 million on them.

No single bug caused it. Four things lined up: dead code left callable, a counter moved without a retest, an old flag given a new meaning, and one server that missed the deployment.

The reused flag

Orders carried a flag, a marker that tells the router how to treat them. In the past, that flag switched on Power Peg. Knight's new code reused the same flag for the new retail program, and was meant to replace Power Peg's unused code.

That plan only works if every server has the new code. On seven servers, the flag meant the new program. On the eighth, the new code never arrived, so the flag still meant Power Peg.

Picture traffic police at eight junctions on one long Mumbai road. Years ago, a sideways palm meant one thing: wave the whole line through until the whistle blows. The force stopped using that drill, but nobody retired the gesture.

Then headquarters gives the old gesture a new meaning: let only the buses go first. Seven constables attend the briefing. The constable at the eighth junction never gets the memo, so he waves every vehicle through.

The missing stop signal

Power Peg used to stop by counting. A counter added up the shares of each parent order that had traded, and the code stopped sending child orders once the parent was filled.

In 2005, Knight moved that counting step to an earlier point in the SMARS code and reused it in another application. It did not retest Power Peg afterwards, so nobody saw that Power Peg no longer stopped.

When Power Peg woke up in 2012, it sent child orders without regard to the shares already filled. Another part of Knight's order system knew the parent orders were filled. SMARS kept sending anyway.

In the junction picture, the whistle that used to stop the eighth constable was moved to a different post years ago. He is waiting for a sound that will never come.

Where the picture breaks: servers do not attend briefings or make choices. Each one simply ran whatever code was on its disk, and the broken counter was a separate fault from the missed copy.

Follow one flagged order through the diagram below. The question box is the whole story: did this server get the new code or not?

The deployment nobody checked

The last link was human. A technician copied the new code to seven servers and missed the eighth. The SEC found that Knight had no written deployment procedure for SMARS and no rule that a second person review the work. The order says a written procedure requiring "a simple double-check" of the deployment could have caught the missed server.

Pause and predict: staff now remove the new code from the seven healthy servers. Orders keep arriving with the same flag. What do those seven servers do next?

Rolling back made it worse because Knight removed the new code while flagged orders kept arriving. The SEC says this caused more incoming parent orders to activate the Power Peg code still present on those seven servers.

A rollback means going back to the previous version of your code. It is usually a safe first move in an incident. Here, the previous version was the dangerous one, because it still read the flag the old way.

Back at the junctions, the control room tells the seven retrained constables to forget the new briefing. It keeps showing the same gesture. Now all eight junctions wave every vehicle through.

The SEC also found that Knight had no written procedures for incident response. So the team hunted for the cause while Knight stayed connected to the market and kept sending orders.

• A rollback changes code, but flags, config and data stay as they are. Check what the old code will do with today's settings.
• When money or customers are leaking, stop the leak first and find the cause second.
• Decide in advance who may pull the kill switch, and write it down, so nobody waits for permission while orders pour out.

No check stopped them because Knight's main order checks sat before SMARS. Inside SMARS, the SEC found no adequate controls against erroneous orders. The tools after it relied on people watching screens. The table lists what Knight had that morning and what the SEC said was missing.

Look at the 33 Account row. That account held positions Knight's systems could not match to a parent order. It had a $2 million gross position limit, a cap on the total size of what it held. But that limit was not linked to any automated control, so the account kept growing.

The price cap had its own gaps: it capped a parent order's price at 9.5 percent beyond the best market price. It did not apply to orders received before the open for the opening auction, the trading that sets each stock's opening price. The SEC order names the 212 orders as examples. It also could not catch errors in prices that moved less than that.

PMON had the same weakness as the 33 Account. At the open, senior staff saw large positions building in the 33 Account. But PMON gave no automatic alerts, did not show the limits and lagged during heavy trading, so people had to notice and act at market speed.

The main rule Knight broke is the SEC's market access rule, Rule 15c3-5, adopted in November 2010. It requires firms with market access to block orders that exceed pre-set credit or capital thresholds. It also requires them to block erroneous orders that break price or size limits.

Knight survived with outside money, then merged. The SEC order says the loss caused significant net capital problems. Net capital is the cash cushion a broker must keep above a legal minimum.

• On 6 August 2012, investors agreed to buy $400 million of Knight's convertible preferred stock, a kind of share that can later turn into ordinary shares.
• On 1 July 2013, Knight's parent company and GETCO combined to form KCG Holdings.
• On 16 October 2013, the SEC censured Knight, a formal reprimand, and fined it $12 million. Knight also had to hire an independent consultant to review its controls.
• The SEC called it its first enforcement action under the market access rule.

The SEC order also describes an earlier warning. In October 2011, after a weekend disaster recovery test, one Knight trading desk kept using test data to send automated price quotes. That cost Knight nearly $7.5 million.

Knight fixed that one system, but did not broadly ask whether its controls could stop erroneous orders from any system. The SEC pointed to that narrow response. Small incidents are cheap lessons only if you learn the general one.

The lessons are about release habits and last-line safety, not clever algorithms. These are our takeaways from the SEC's findings.

• Delete retired code. If a feature is off for good, remove the code and its flag in the same change.
• Never give an old flag a new meaning. A new feature gets a new flag name, so old code cannot misread it.
• Automate deployments, then verify. Every server must report the new version before any flag is turned on.
• Make early warnings loud. An error that appears before the open is a reason to delay, not to ignore.
• Put hard limits at the exit. The last component before the market must block overfills and firm-wide limit breaches by itself.
• Have a kill switch and a written plan. Name who can pull it, and when, before you ever need it.

Beginners can stop here. The rest shows the two checks the team adds to its own router, with code you can run.

What the team changes before it ships

The team goes through the list one line at a time. The new feature gets a fresh flag, split_v2. The same release deletes use_sweep and the retired feature's code. Then a script replaces copying by hand.

The script deploys to all six servers, then asks each one which version it runs. Here is the checking part, with a pretend fleet at the bottom where one server missed the deploy. In real use, get_version would ask each server over the network.

EXPECTED = "router-7.2"

def check_rollout(servers, get_version):
    wrong = {}
    for host in servers:
        try:
            found = get_version(host)
        except OSError:
            found = "no answer"
        if found != EXPECTED:
            wrong[host] = found
    if wrong:
        raise SystemExit(f"Keep the flag off. Wrong code on: {wrong}")
    print(f"All {len(servers)} servers run {EXPECTED}.")

# Try it on a pretend fleet of six servers where one missed the deploy.
fleet = {f"order-{n}": EXPECTED for n in range(1, 7)}
fleet["order-6"] = "router-7.1"
check_rollout(list(fleet), fleet.get)

Save it as check_rollout.py and run python3 check_rollout.py. We ran it on Python 3.13: it prints one line to the error stream naming order-6 and router-7.1. On Linux or macOS, echo $? then prints 1, the exit code that tells a release pipeline to stop.

Two lines matter most. The except branch treats a server that does not answer as wrong, because silence is not proof. The raise stops the release script, so the flag can only go on after every server passes.

The second change sits right before the exchange. Knight's gap was that nothing compared what left the router with what came in. The team's guard does exactly that, plus a firm-wide limit of Rs 50 crore (500 million rupees) in open orders.

from dataclasses import dataclass

FIRM_LIMIT = 500_000_000   # Rs 50 crore of open orders

@dataclass
class Parent:
    id: str
    total: int      # shares the customer asked for
    sent: int = 0   # shares already sent as child orders

class OrderGuard:
    def __init__(self, halt):
        self.open_value = 0
        self.halted = False
        self.halt = halt          # pulls the kill switch

    def stop(self, reason):
        self.halted = True
        self.halt(reason)

    def allow(self, parent, qty, price):
        if self.halted:
            return False
        if parent.sent + qty > parent.total:
            self.stop(f"Parent {parent.id} would be overfilled")
            return False
        if self.open_value + qty * price > FIRM_LIMIT:
            self.stop("Firm-wide limit reached")
            return False
        parent.sent += qty
        self.open_value += qty * price
        return True

# Replay a runaway: a buggy router tries 50 child orders of 100 shares
# for a parent that wants only 1,000 shares. Here halt just prints.
guard = OrderGuard(halt=print)
p = Parent("P1", total=1_000)
allowed = sum(guard.allow(p, 100, 2_500) for _ in range(50))
print("child orders allowed:", allowed, "shares sent:", p.sent)

Run python3 order_guard.py. The demo prices each share at Rs 2,500. It prints Parent P1 would be overfilled once, then child orders allowed: 10 shares sent: 1000. The guard let exactly 1,000 shares out of 5,000 attempted, then refused everything.

What just happened? The first check counts shares already sent for each parent, not just shares filled, so it stops before an overfill, not after. Once it halts, the guard refuses every later order, like a kill switch.

The second check caps the whole firm, which no single account limit can do. A real guard would also lower open_value when orders fill or are cancelled, and would run in the same process that writes to the exchange.

Here is the team's new release path. Notice that the flag sits behind the version check, and the guard sits between the flag and the exchange.

The team adds one more thing: a single page that says when to pull the kill switch. If the guard halts, or positions grow faster than expected, the person on duty pulls it first and asks questions later.

Tell it as a chain of small failures, then turn each link into a design rule. Strong answers spend most of their time on the controls that would have stopped it.

1. 1

   Give the one-line summary

   A missed server plus a reused flag woke up dead code in a trading router, which sent millions of orders in about 45 minutes.

2. 2

   Name the latent causes

   Retired code left callable since 2003, a fill counter moved in 2005 without a retest of Power Peg, and an old flag given a new meaning.

3. 3

   Name the trigger

   A staged deployment that missed one of eight servers, with no second person and no automatic version check.

4. 4

   Explain why it spread

   No automatic limit at the router's exit, warnings sent as ignored emails, and a rollback that put old code on the healthy servers.

5. 5

   Propose the fixes

   Delete dead code, use new flag names, verify every server before enabling a flag, and add a pre-send guard with a kill switch.

6. 6

   Generalise

   Say how the same guard applies to any system that sends payments, messages or orders, and where you would put it.