The 2021 Facebook Outage Explained: BGP and DNS

Every name the company owns stops working, even while its servers keep running. Phones cannot turn the name into an address, so apps fail everywhere at once.

That is what happened on Monday, 4 October 2021, when Facebook, Instagram and WhatsApp disappeared for roughly six hours. People still search for it as the Facebook outage 2021.

The cause was not one failure but two. A routine command cut Facebook's backbone. Then a well-meant safety rule inside its DNS servers turned a broken company into an invisible one.

Picture a common setup. A regional internet provider in Assam runs a small backbone joining two data centres. Three DNS sites answer for every website it hosts. The setup is made up, but the rule it runs is the real one.

Each DNS site follows Facebook's rule: if I cannot reach a data centre, withdraw my route. The provider has backbone maintenance planned for next week. This guide follows Facebook step by step, then fixes the provider before that window.

A command meant to check spare backbone capacity took down every backbone connection instead, and a bug in the audit tool let it run. The DNS servers then withdrew their routes, so no Facebook name worked anywhere.

Nobody broke in. Facebook's parent company, now called Meta, said the root cause was a faulty configuration change on its own side. It also said there was no evidence that user data was compromised.

Meta published two posts on its engineering blog. The first, on 4 October, blamed configuration changes on the backbone routers that carry traffic between data centres. The second, on 5 October, explained the full chain.

Here is that chain in four short steps:

• During a routine maintenance job, a command meant to assess global backbone capacity took down all the connections in the backbone.
• An audit tool, built to stop dangerous commands like this one, had a bug and did not stop it.
• Cut off from the data centres, Facebook's DNS servers disabled their BGP advertisements, the messages that tell the internet where they are.
• The rest of the internet could no longer find those DNS servers, so no Facebook name could be looked up.

The first step broke Facebook. The third step made Facebook vanish. That difference is the heart of this outage.

You need five: prefix, autonomous system, BGP, DNS and TTL. BGP finds the way to an address, and DNS finds the address for a name. Opening any website needs both.

IP address and prefix. An IP address is a number that identifies a machine on the internet, like 203.0.113.10. A prefix is a block of addresses written with a slash. For example, 203.0.113.0/24 holds 256 addresses.

Autonomous system. An autonomous system (AS) is a set of routers under one technical administration, with its own number. Facebook's network is AS 32934.

BGP. The Border Gateway Protocol lets autonomous systems swap reachability news. A network announces (advertises) a prefix to say it can deliver traffic there. It withdraws the prefix to say it no longer can, and other routers drop that path.

DNS. The Domain Name System turns names into addresses. An authoritative DNS server holds the true answers for a company's names. A resolver, usually run by your internet provider or a public service, asks on your behalf and caches the answers.

TTL. Time to live is the number of seconds a resolver may keep a cached answer. When it runs out, the resolver must ask the authoritative server again. In our setup, the provider gives its website records a TTL of 300 seconds.

Here is a picture from Assam. Ferries cross the Brahmaputra to Majuli, the river island. At each mainland ghat (riverside landing), a board lists today's crossings. That board is BGP: it tells travellers which routes exist right now.

On the island, a directory desk at each jetty tells visitors which village and lane a family lives in. That desk is DNS. You need the board to reach the desk, and the desk to find the house.

Where the picture breaks: there is no single board. Every BGP router keeps its own copy of the routes, learned from its neighbours, so a withdrawal spreads from router to router.

Now the normal path. Follow the arrows from your phone. Notice that the resolver can reach Facebook's DNS servers only because a BGP route points to them.

The outage ran from about 15:40 UTC on 4 October 2021 to about 21:30 UTC. That is 9:10 pm to about 3 am India time (IST). UTC is the world's reference clock.

The clock times come from Cloudflare, which runs the public resolver 1.1.1.1 and watched it happen from outside.

Cloudflare checked two of Facebook's DNS prefixes, 185.89.218.0/23 and 129.134.30.0/23, on a public route server. Neither was in its table. Other Facebook addresses stayed routed. They were of little use, because nothing could look up a name to find them.

A maintenance command meant to assess global backbone capacity took down every backbone connection by mistake. An audit tool failed to block it because of a bug. Meta says the outage was triggered by the system that manages backbone capacity.

Big networks do maintenance every day. Engineers take a link or router out of service, fix or upgrade it, and put it back. To keep this safe, Facebook's systems check commands before they run.

That check is the audit tool. Its job is to look at a command and refuse it if it could do serious harm. This time it had a bug, and the command went through.

Why a capacity check could cut every link

Meta's posts do not publish the exact command, as of 19 October 2026, so nobody outside knows its wording. What matters for you is the pattern. One command could reach every backbone connection at once.

Engineers call this the blast radius: how much breaks if one action goes wrong. Meta names one safeguard that should have stopped it, the audit tool, and a bug let the command through. Tools have bugs.

Facebook's DNS servers were set to disable their BGP advertisements whenever they could not speak to the data centres. When the backbone died, every DNS site met that rule almost at once, and all of them disappeared together.

The rule makes sense for one site. Meta explains that losing contact with the data centres is treated as a sign of an unhealthy network connection. A site with bad links steps aside, so users reach a healthier site.

It fails when every site loses the data centres at once. Then each site steps aside, and no site is left. In Meta's words, the DNS servers became unreachable even though they were still operational.

Back on Majuli. Each directory desk has a rule: if word stops coming from the island's villages, ask the ghats to take its route off the board. Then one monsoon night, the island's inner roads flood.

Every desk loses touch with the villages, and every desk asks for its route to come down. Each desk is open, staffed and correct. But no board on the mainland shows a ferry to any of them.

Follow the chain from the maintenance command. The question box is the DNS rule. Notice that every DNS site reached the same No answer at once.

SERVFAIL is the DNS error code for a server failure. A resolver sends it back when it could not get an answer, here because no authoritative server replied. Cloudflare says 1.1.1.1 and other big public resolvers started returning, and caching, SERVFAIL for Facebook's names.

Cached answers did not help for long. Once an answer's TTL ran out, the resolver had to ask Facebook's DNS servers again, and found no path.

Then the retries started. Cloudflare says apps do not accept an error for an answer and retry, sometimes aggressively, and people keep reloading. It wrote that resolvers worldwide were handling 30 times more queries than usual.

Cloudflare warned this could cause delays and timeouts on other platforms. Even so, it says the vast majority of its own DNS requests kept resolving in under 10 milliseconds.

Recovery was slow because the ways into the network ran over the network that was broken. Engineers had to reach the data centres in person, and strong physical security slowed them down.

Normally, engineers fix routers remotely. Many companies also keep an out-of-band network: a separate path used only to manage equipment when the main network fails. Meta says both its primary and out-of-band access were down.

The loss of DNS hurt inside the company too. Meta says the total loss of DNS broke many of the internal tools engineers would normally use to investigate and fix an outage.

So people went on site. Meta says the data centres are hard to get into, and their routers are designed to be hard to change even with physical access. Those protections added time before anyone could fix the backbone.

The report's author, Santosh Janardhan, defends the choice. He writes that much stronger security every day is worth a slower recovery from a rare event like this one.

Here is the path the recovery had to take. Each blocked shortcut pushed the team one step further out, from a keyboard to a car to a locked door.

Facebook brought services back carefully, managing the rising load instead of switching everything on at once, because a sudden surge could cause fresh failures. Its regular failure drills helped with this part.

Think about what a restart means at this size. Meta says individual data centres reported dips in power use of tens of megawatts. Reversing that dip in one moment could put electrical systems at risk.

Caches are the other risk Meta names. A cache is a fast store of recent answers that shields slower systems behind it. A sudden rush of returning users can overload caches and everything behind them.

Meta runs what it calls storm exercises: drills that take a service, a data centre or a whole region offline on purpose. Those drills helped the restart. But Meta says it had never run one that took its global backbone offline.

Meta did not publish its exact restart steps. Here is a generic staged restart, not Meta's runbook:

1. 1

   Restore the network first

   Bring backbone links back and confirm the DNS prefixes are announced again, as seen from outside your own network.

2. 2

   Let DNS answer again

   Resolvers start getting answers instead of SERVFAIL. Their caches fill as users ask, so the retry storm dies down.

3. 3

   Open traffic one slice at a time

   Let in a share of users per region, then raise it while watching error rates.

4. 4

   Watch power and caches

   Check that power draw and cache hit rates climb smoothly, not in one jump.

5. 5

   Pause before the next slice

   Hold if anything wobbles. A second crash during recovery costs far more than a slow restart.

Make each site ask its peers before stepping aside. A site should withdraw only while another site still reaches the data centres. If nobody does, it should stay announced and page a human.

First, the setup behind it. The provider announces 203.0.113.0/24 from all three DNS sites at once. This is anycast: one address served from many places, with routers sending each user to the nearest copy.

RFC 4786, the best-practice guide to running anycast, says tying the route to the health of the service behind it is desirable. The open question is what counts as unhealthy.

Below is a small simulation that needs only Python 3 (we ran it on 3.13). It compares the old rule with the peer-aware one, on two failures: one site cut off, and the whole backbone down.

CORE = {"dc-1", "dc-2"}  # the two data centres

class Site:
    def __init__(self, name, reach):
        self.name = name
        self.reach = reach  # data centres this DNS site can reach right now
        self.announced = True

    def core_ok(self):
        return bool(self.reach & CORE)

def old_rule(site, peers):
    # Withdraw whenever this site cannot reach a data centre.
    site.announced = site.core_ok()

def new_rule(site, peers):
    if site.core_ok():
        site.announced = True
    elif any(p.core_ok() for p in peers):  # asked over the public internet
        site.announced = False  # only my links are bad: step aside
    else:
        site.announced = True  # core lost for everyone: stay up
        print("PAGE on-call:", site.name, "and every peer lost the core")

def run(rule, reach):
    sites = [Site(name, r) for name, r in reach.items()]
    for s in sites:
        rule(s, [p for p in sites if p is not s])
    return [s.name for s in sites if s.announced]

one_site_cut = {"dns-a": set(), "dns-b": CORE, "dns-c": CORE}
backbone_down = {"dns-a": set(), "dns-b": set(), "dns-c": set()}
print("old rule, one site cut:", run(old_rule, one_site_cut))
print("old rule, backbone down:", run(old_rule, backbone_down))
print("new rule, one site cut:", run(new_rule, one_site_cut))
print("new rule, backbone down:", run(new_rule, backbone_down))

Pause and predict: under the old rule, how many sites stay announced when the backbone dies? Run it with python3 health_rule.py. You should see exactly this:

old rule, one site cut: ['dns-b', 'dns-c']
old rule, backbone down: []
new rule, one site cut: ['dns-b', 'dns-c']
PAGE on-call: dns-a and every peer lost the core
PAGE on-call: dns-b and every peer lost the core
PAGE on-call: dns-c and every peer lost the core
new rule, backbone down: ['dns-a', 'dns-b', 'dns-c']

What just happened? The old rule handled a single broken site well, but withdrew all three when the backbone failed. That empty list is Facebook's outage in miniature.

The peer-aware rule gave the same answer for one broken site. When everyone lost the core, it kept all three sites announced and paged the on-call engineer three times.

Where the peer-aware rule breaks

The peer question must travel over the public internet, never the backbone, or it fails with the thing it checks. The simulation reads each peer's state directly. In a real system, treat a peer you cannot reach as unhealthy.

That choice has a cost. A truly broken site that also cannot reach any peer stays announced, and keeps receiving users. For DNS, a degraded site usually beats no site at all.

Staying announced does not bring hosted websites back if the data centres are cut off. But names still resolve, the provider's own tools that use those names keep working, and resolvers stop retrying in a storm.

This is one design, not Meta's published fix. As of 19 October 2026, Meta's posts do not describe how it changed the DNS rule.

The provider also adds 4G modems from a different mobile carrier at both data centres, each with its own battery backup. This gives on-call an out-of-band path that shares no fibre, power or DNS with the provider's own network.

Before the maintenance window, the team runs a drill on a quiet Sunday. They cut a lab copy of the backbone and check three things: DNS still answers, on-call gets paged, and the 4G login works.

Not for Facebook, because the data centres behind the names were cut off too. But a resolver setting called serve-stale can keep names answering when only the DNS servers are unreachable.

Serve-stale is defined in RFC 8767 (March 2020). A resolver may answer with expired cached data when it cannot reach the authoritative servers to refresh it. The RFC recommends giving such stale answers a TTL of 30 seconds.

We tested this on Unbound 1.19.2, an open-source resolver from NLnet Labs, against a tiny local DNS server. The record had a TTL of 5 seconds, and then we switched the server off.

With the setting off, the resolver had nothing to give once the TTL ran out. With it on, it handed back the old address at once, marked with a TTL of 30 seconds.

Check yours with unbound-checkconf -o serve-expired. On 1.19.2 it prints no. The related limit, serve-expired-ttl, defaults to 0 there, meaning no limit. The current Unbound manual lists 86400 seconds (one day), so check your version. That manual also sets serve-expired-client-timeout, the wait before serving stale data, to 1800 milliseconds. On 1.19.2 it is 0.

Serve-stale has limits. It only helps names already in the cache, and it can keep sending users to an address that is really gone. We do not know which public resolvers used it in 2021.

The main lessons are to limit what one command can touch, and to stop health checks from removing every copy of a service at once. Also keep a way in that shares nothing with the main network.

• Limit the blast radius. Run network-wide changes region by region, with a pause and an automatic check between steps.
• Do not trust one guard. Add a second layer, such as a dry run that shows exactly what a command will change.
• Make health rules count their peers. A site should step aside only while other sites are still healthy.
• Keep a separate way in. Out-of-band access should not share links, power or DNS with the network it rescues.
• Write down the break-glass path, the emergency way in. Know who can enter which building at 2 am, with which pass, and practise it.
• Restart in slices. Plan how load returns before the day you need it.

Tell it as a chain of three links: a bad change cut the backbone, a health rule withdrew DNS everywhere, and recovery needed the broken network. Then turn each link into a design lesson.

1. 1

   Open with one sentence

   A backbone command cut off the data centres, and DNS servers withdrew their BGP routes, so the whole company vanished from the internet.

2. 2

   Define BGP and DNS briefly

   BGP tells the internet which addresses a network can reach. DNS turns names into addresses. Point out that DNS itself needs a BGP route.

3. 3

   Name the amplifier

   The DNS health rule was sensible for one site and fatal for all of them at once. Call it a correlated failure: many copies failing together from one shared cause.

4. 4

   Explain the slow recovery

   Remote and out-of-band access shared the broken network, internal tools needed DNS, and data centre security took time.

5. 5

   Propose fixes

   Staged changes with automatic checks, peer-aware health rules, truly separate out-of-band access, and restarts in slices.

6. 6

   State the tradeoff

   Strong physical security slows recovery. Say you would keep it, and rehearse the break-glass path instead.