Disaster Recovery: RTO, RPO and Multi-Region Planning

If your app runs in one cloud region and that region fails, your app stays down until the region recovers or you rebuild somewhere else. Without disaster recovery planned in advance, that rebuild can take days.

A region is a cloud provider's group of data centres in one area, such as Mumbai. Each region holds a few availability zones. A zone is one or more separate data centres with their own power, cooling and network.

Picture a common setup. A pension-records service keeps the records of 18 lakh (1.8 million) members and pays a monthly pension to 3.2 lakh retired people on the 1st of each month. The setup is made up, but the problem is real.

Everything runs in one region in Mumbai, spread across three availability zones, with PostgreSQL as the main database. If one data centre fails, the other two carry on, so the team feels safe.

Then someone in leadership reads about 19 and 20 October 2025. A whole AWS region, us-east-1 in Northern Virginia, had problems for about 14 and a half hours, from 11:48 PM to 2:20 PM US Pacific time.

AWS's own summary says a hidden timing bug, a race condition, in the automation that manages DynamoDB's DNS records left the database service's address empty. DNS is the internet's phone book, turning names into addresses.

Programs could not find DynamoDB. Services that depend on it had trouble too. Launches of new EC2 virtual servers ran into errors for hours, although servers already running stayed healthy.

The question to the team is simple. If Mumbai went down like that on the 1st of the month, when do pensioners get paid? Nobody has a good answer.

There is one full backup each night, stored in the same Mumbai region. Rebuilding every server by hand and restoring 2 TB of data would take two or three days. This guide builds the team a real plan instead.

Disaster recovery (DR) is the plan, the copies and the practised steps that bring a system and its data back after a large failure. It covers failures that your everyday redundancy cannot absorb.

High availability and disaster recovery are different jobs. High availability keeps the app running through small failures, like one server or one zone dying. Disaster recovery handles the big ones: a region outage, ransomware (criminals locking your data) or a deleted database.

A few words come up in every DR plan:

• Backup: a copy of data frozen at one moment, stored somewhere else, that you can restore later.
• Replica: a second copy of a database kept up to date continuously, usually a moment behind the main one, called the primary.
• Primary region and DR region: the region that normally serves users, and the one you recover into. Here, Mumbai and Hyderabad.
• Failover: moving live traffic from the broken site to the recovery site. Failback: moving it home again once the original site is healthy.

Think of a family in Chennai getting ready for the monsoon. Property papers on the top shelf survive if water enters one room. If the whole street floods, the top shelf goes under too.

So careful families keep photocopies at a relative's house in another city, say an aunt in Coimbatore. The same flood will not reach both houses. Availability zones are the top shelf. A second region is the aunt's house.

Where the picture breaks: a flood never copies itself. A software mistake does. A bad delete in Mumbai reaches a live copy in Hyderabad within seconds, which is why the later section on backups matters.

RTO (recovery time objective) is how long a system may stay down after a disaster. RPO (recovery point objective) is how much recent data you can afford to lose, measured as time before the disaster.

NIST, a US standards agency, defines RTO as the longest a system resource can stay unavailable before the impact becomes unacceptable. It defines RPO as the point in time before an outage to which data must be recovered.

NIST adds a third number, maximum tolerable downtime (MTD): the total outage the business owner will accept, all effects included. It says the RTO must normally be shorter than the MTD, so the outage never runs past it.

That gap is real work. After systems return, people still check the data and catch up on what was missed.

Back to the photocopies at the aunt's house. RPO is how old those copies are. If you last copied them in March, anything signed after March is gone. RTO is how long before the family lives normally again.

The timeline below shows both numbers for an imagined Mumbai failure on the payouts database. Read it left to right. The data loss sits before the failure, and the downtime sits after it.

You buy the two numbers in different ways. You shrink RPO by copying data more often, or continuously. You shrink RTO by keeping more of the recovery site ready and by rehearsing the steps.

You choose them by asking what an hour of downtime and an hour of lost data would cost each service, then paying only for what that justifies. The numbers come from the business, not from engineers.

The team spends an afternoon with the operations head and the compliance team. For each service they ask two questions. What breaks for members if this is down for an hour? What breaks if the last hour of changes vanishes?

• Pension payouts. A missed payment hurts retired people who depend on it, and a lost change could undo a bank account update. Targets: RTO 1 hour, RPO 5 minutes.
• Member portal. Members can wait a few hours to download a statement. Targets: RTO 4 hours, RPO 15 minutes.
• Reports and analytics. These are rebuilt from the main data every night. Targets: RTO 24 hours, RPO 24 hours.

Grouping services into tiers like this stops you paying top prices to protect a report nobody reads until Monday. The team writes the targets into one small file in the code repository.

Monitoring and the runbooks (step-by-step recovery instructions) read that same file, so nobody works from an old spreadsheet.

# dr-plan.yaml: recovery targets for each service
services:
  payouts:
    tier: 1
    rto_minutes: 60
    rpo_minutes: 5
    strategy: warm_standby
  member_portal:
    tier: 2
    rto_minutes: 240
    rpo_minutes: 15
    strategy: pilot_light
  reports:
    tier: 3
    rto_minutes: 1440
    rpo_minutes: 1440
    strategy: backup_and_restore

The strategy line is the interesting one. Each tier recovers in a different way, and the next section explains the four choices.

The four common strategies are backup and restore, pilot light, warm standby and multi-site active-active. Each keeps more of the DR region running, so each costs more and recovers faster.

AWS's disaster recovery whitepaper uses these four names, as of 19 October 2026. The idea is older than the cloud: NIST describes cold, warm and hot alternate sites for physical buildings in much the same way.

Backup and restore

You copy backups to the DR region and build everything else only after a disaster. It is the cheapest choice. Recovery takes hours or longer, because servers, networks and databases must be created and data restored first.

To rebuild quickly, keep your whole setup as infrastructure as code: files that describe servers and networks, which a tool turns into real resources. Without it, your team rebuilds from memory, under stress, at night.

Pilot light

Data is copied continuously to the DR region, and the core, such as the database, is always on. Application servers are defined but switched off. During a disaster, you switch them on and scale them up.

Warm standby

A smaller but complete copy of production runs in the DR region all the time. The AWS whitepaper puts the difference plainly: warm standby can handle traffic at once, at reduced capacity, and pilot light cannot until you act first.

Multi-site active-active

Both regions serve real users all the time. If one fails, the other is already running, so recovery time can be close to zero. It is the most complex and costly choice, mostly because both regions may accept writes.

Writes in two places can clash. Say a member changes a nominee in Mumbai while a clerk edits the same record in Hyderabad. Something has to decide which change wins. The CAP theorem says a system cut in two must pick consistency or availability. PACELC adds that, even when healthy, it trades consistency against speed.

The team picks a strategy per tier, not one for everything. Payouts get warm standby, the portal gets pilot light, and reports get backup and restore. The extra bill is far below running every service active-active.

Here is the design for payouts. Follow the path users take today, then find the parts in Hyderabad that wait for a bad day.

Async (asynchronous) replication means Mumbai confirms a write first and ships it to Hyderabad a moment later. Writes stay fast, but changes still in flight are lost if Mumbai dies. PostgreSQL's docs say that loss grows with the replication delay at failover.

Replication copies every change, including the bad ones, so it cannot undo a mistake. In our PostgreSQL 16.15 test, a delete of 600 rows on the primary was on the replica within one second.

GitLab learned this in public on 31 January 2017. Replication to its secondary database had broken, and an engineer working to re-sync it wiped the database directory. The command ran on the primary instead, and about 300 GB of data was gone.

The replica could not help, and the regular backups had been failing silently. The backup job used pg_dump 9.2, PostgreSQL's tool for exporting a database to a file, against a PostgreSQL 9.6 database, and its failure emails were being rejected. GitLab was not using WAL archiving either.

It restored from a disk snapshot taken 6 hours before the outage, and the site was down for about 18 hours. GitLab estimated it lost at least 5,000 projects, 5,000 comments and about 700 users. Its postmortem says nobody owned testing the backup procedure.

The fix is point-in-time recovery (PITR): restoring the database to its exact state at a moment you choose, such as one second before the bad delete. PostgreSQL does this with a base backup, a full copy of the database files, plus archived WAL.

WAL, the write-ahead log, is PostgreSQL's running diary of every change. Continuous archiving copies each finished WAL file to safe storage. To recover, you restore the base backup and replay the diary up to the moment you name.

PostgreSQL archives only finished WAL files, so a quiet database can leave recent changes unarchived for a long time. The archive_timeout setting forces a new file at least that often, when there has been any activity. The docs call a minute or so usually reasonable.

The team sets it to 60 seconds, so the archive path alone gives an RPO of about a minute plus copy time. The archive goes to versioned storage in Hyderabad, which keeps old copies of every file, under a separate cloud account.

A script gone wrong in the Mumbai account, or stolen Mumbai credentials, cannot delete that archive. That holds only while the Mumbai account has no delete rights in the backup account.

Failover makes the DR region the new primary. You stop the old primary taking writes, promote the replica, and point users at the new region. The order of those steps matters more than their speed.

The first decision is who presses the button. The AWS whitepaper warns that automatic failover on health checks needs caution, because a false alarm still costs data and downtime. It notes that failover is therefore often started by hand, with every step after that still automated.

The team's rule is simple: two named people agree, then one command does the rest. In the diagram, the question box is where people decide; everything after it runs from a tested script.

Fencing means making sure the old primary can no longer accept writes. Skip it, and Mumbai may come back half-alive and take writes while Hyderabad does too. That is called split brain, and the next section shows it happening.

PostgreSQL does not detect a failed primary or fence it for you. Its docs say so, and point to outside tools for detection and IP address moves. Your failover script, or a tool you choose, has to do the fencing.

DNS records carry a TTL (time to live), which tells resolvers how long to cache an answer. The team keeps 60 seconds on its main name, then measures in each test how long real traffic actually takes to move.

Fail over with data plane actions

The AWS whitepaper separates the data plane, which serves live traffic, from the control plane, which creates and changes resources. It advises using only data plane actions during failover, because data planes usually have higher availability goals.

The same page lists Auto Scaling and restoring from backup as control plane actions. So the scale-up step is the weak link, even in warm standby. October 2025 showed the risk, when launches of new servers in the broken region ran into errors for hours.

The team sizes the two always-on Hyderabad servers to pay pensions at reduced speed, before any scale-up succeeds. It also checks service quotas, the limits on how much it may run, in Hyderabad ahead of time.

You get two primaries that both accept writes, and their data starts to differ at once. We ran this on PostgreSQL 16.15 with a local primary and an async replica.

Pause and predict: the old primary was never told it lost its job. What does it do with the next write that reaches it?

Before promotion, the replica refused writes with "cannot execute INSERT in a read-only transaction". After pg_promote(), the replica accepted writes. The old primary, never fenced, kept accepting them too.

We wrote row 5000 on both. The old primary stored amount 222 and the new primary stored 111. The new primary had moved to timeline 2, PostgreSQL's label for a new history branch, while the old one stayed on timeline 1.

PostgreSQL's failover docs say two servers that both think they are primary lead to confusion and ultimately data loss. They call the fix STONITH (shoot the other node in the head). In a cloud, you can fence by cutting the old primary's network access or stopping it before you promote.

Failback with pg_rewind

To come home, you turn old Mumbai into a replica of Hyderabad. The pg_rewind tool finds where the two histories split and copies the data blocks changed since then, plus some whole files such as configuration, instead of the whole database.

It has a catch you must handle before the disaster. pg_rewind needs wal_log_hints = on or data checksums on the old primary, and neither is on by default in PostgreSQL 16. In our test without them it stopped with this error:

pg_rewind: error: target server needs to use either data checksums or "wal_log_hints = on"

With wal_log_hints = on, pg_rewind printed "Done!" and the old primary rejoined as a replica. Row 5000 then read 111: the stray write of 222 was thrown away. Save any split-brain writes you need before rewinding.

Our first attempt also failed for a second reason. The old primary no longer had the WAL files from the point where the histories split, so we kept more WAL with wal_keep_size. Plan where that WAL will come from; pg_rewind's --restore-target-wal option can fetch it from your archive.

One more surprise: pg_rewind copied the new primary's configuration files over the old one's, its port setting included. We had to set the port back before the old primary would start as a replica. The pg_rewind docs warn about this.

With synchronous replication, each commit waits until a named standby confirms it has the change. If that standby crashes, commits keep waiting, and PostgreSQL's docs warn they may never complete. A broken link should look the same to the primary, though we only tested a stopped standby.

That is the price of an RPO close to zero across regions. We ran it on PostgreSQL 16.15 with one synchronous standby named in synchronous_standby_names, then stopped the standby to imitate a broken link to the DR region.

• Setting statement_timeout to 2 seconds did not end the wait. In the PostgreSQL 16 source, that timeout is switched off just before the commit step, where the wait happens. Our outer timer killed the client after 8 seconds, and its server session kept waiting. The PostgreSQL 16 documentation pages we read do not mention this, as of 19 October 2026, so test it on your version.
• Another commit was still waiting 5 seconds later, shown in pg_stat_activity with wait_event SyncRep.
• While it waited, other sessions could not see the row.
• Cancelling the wait with pg_cancel_backend made the row visible, with a warning that the transaction had already committed locally but might not be on the standby.

So a cancelled wait does not undo the write; the client gets success with only a warning. If your app gives up on its own timer instead, it shows the user an error for a payment the primary keeps once the wait ends. Plan retries that check before writing again.

Clearing synchronous_standby_names and reloading the configuration let new commits flow again in under a second. That step trades the zero-RPO promise for availability, so decide in advance who may take it.

You check by restoring for real and by measuring lag in bytes, not only in seconds. Both checks below ran on PostgreSQL 16.15, and the outputs shown are what we saw.

Run a point-in-time restore

1. 1

   Take a base backup

   Run pg_basebackup against the primary while WAL archiving is on. Keep the backup and the archive outside the primary region.

2. 2

   Note the target time

   Pick a moment just before the mistake, for example from the application log or from SELECT now() taken before a risky change.

3. 3

   Configure recovery

   In a fresh copy of the backup, set restore_command, recovery_target_time and recovery_target_action = 'promote', then create an empty recovery.signal file.

4. 4

   Start and confirm

   Start the server. When pg_is_in_recovery() returns f, count the rows you care about and compare them with what you expect.

In our test, 1,500 rows existed before a bad delete. The restored copy had all 1,500, recovery.signal had been removed, and the log showed these lines:

LOG:  starting point-in-time recovery to 2026-10-21 19:57:25.765651+00
LOG:  recovery stopping before commit of transaction 733, time 2026-10-21 19:57:26.812241+00
LOG:  archive recovery complete

What just happened? PostgreSQL replayed the archived WAL on top of the base backup and stopped just before the delete committed. The restore took 2.4 seconds, but that was a tiny test table. Your real restore time, measured on production size, is the number that belongs next to your RTO.

Also check the archive is moving. On the primary, pg_stat_archiver shows how many files were archived and how many failed:

SELECT archived_count, last_archived_wal, failed_count FROM pg_stat_archiver;
-- our output: 4|000000010000000000000003|0

Each archived WAL segment file was 16,777,216 bytes (16 MiB), even ones closed early by a forced switch. The docs confirm early files keep full length, so a very short archive_timeout bloats storage.

Measure replica lag the right way

On the primary, ask how many bytes the replica still has to replay. On the replica, PostgreSQL can tell you how long ago it last replayed a transaction.

-- on the primary
SELECT application_name, pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS replay_lag_bytes, replay_lag FROM pg_stat_replication;
-- our output: dr_replica|0|00:00:00.057498

-- on the replica
SELECT pg_is_in_recovery(), now() - pg_last_xact_replay_timestamp() AS since_last_replayed;
-- our output: t|00:00:01.058365

The replica-side number misleads on a quiet database. In our test, with no writes for a few seconds, it read 7 seconds while the primary showed 0 bytes left to replay. Nothing was missing.

So alert on bytes behind, or on the primary's replay_lag, and treat time since last replay only as a hint. Our test replica also reported sync_state async, PostgreSQL's default.

One catch, from the PostgreSQL monitoring docs: replay_lag turns NULL a short while after an idle replica has fully caught up. The docs leave it to you how to show that NULL; together with 0 bytes behind, it means the replica is caught up.

The check below turns those numbers into alerts against the plan file. Monitoring helpers, not shown, supply the lag in seconds from the primary's replay_lag, and the times of the last archived WAL file and last successful restore test.

from datetime import datetime, timedelta, timezone
import yaml

with open("dr-plan.yaml") as f:
    PLAN = yaml.safe_load(f)["services"]

def dr_alerts(service, replica_lag_s, last_wal_archived, last_restore_test):
    rpo_s = PLAN[service]["rpo_minutes"] * 60
    now = datetime.now(timezone.utc)
    alerts = []
    if replica_lag_s > rpo_s / 2:                  # warn at half the RPO
        alerts.append(f"{service}: replica {replica_lag_s}s behind")
    if (now - last_wal_archived).total_seconds() > rpo_s:
        alerts.append(f"{service}: WAL archive older than the RPO")
    if now - last_restore_test > timedelta(days=7):  # weekly restore test
        alerts.append(f"{service}: no successful restore test this week")
    return alerts

now = datetime.now(timezone.utc)
print(dr_alerts("payouts", 200, now - timedelta(minutes=8), now - timedelta(days=9)))

Run with Python 3.13 and PyYAML 6.0.3, it prints all three alerts for payouts: replica 200s behind, WAL archive older than the RPO, and no successful restore test this week. Healthy inputs print an empty list.

The lag alarm fires at half the RPO, so the team hears about trouble while there is still time. The restore-test alarm turns GitLab's lesson into a rule: an untested backup counts as a problem.

You test it by doing the recovery for real on a schedule: restoring backups, failing services over, and timing each step against your RTO and RPO. NIST's contingency planning guide makes plan testing, training and exercises one of its seven planning steps.

1. 1

   Every week: restore a backup

   A script restores the latest base backup plus WAL into a scratch database, checks row counts and records how long the restore took.

2. 2

   Every month: walk through the runbook

   The on-call team reads the failover steps aloud without touching anything. Anyone confused by a step rewrites it on the spot.

3. 3

   Every quarter: fail over for real

   On a quiet Sunday, payouts move to Hyderabad for four hours, then back. The team measures the actual RTO and RPO.

4. 4

   After every test: fix and record

   Each gap becomes a ticket with an owner. The measured numbers go into the plan file, next to the targets.

In this setup, the first real failover test takes 2 hours 40 minutes against a 1-hour target (made up). Most of that time goes on hunting for a database password stored only in Mumbai's secret store.

The team copies the secrets to Hyderabad, and the next test takes 48 minutes. Data alone is never enough for a recovery.

Tests also catch slow drift. Someone adds a new service in Mumbai and forgets Hyderabad. The database doubles in size, and the restore quietly takes twice as long. Regular tests reveal these before a real disaster does.

Start from the business numbers, choose a strategy per service, and show that you know replication alone does not protect against data mistakes.

1. 1

   Ask what a disaster means here

   One server, one zone, a whole region or a bad delete? Each needs a different answer, so agree on the scope first.

2. 2

   Get RTO and RPO per service

   Ask what an hour of downtime and an hour of lost data would cost. Group the services into tiers.

3. 3

   Pick a strategy per tier

   Backup and restore, pilot light, warm standby or active-active. Say the cost of each choice out loud.

4. 4

   Draw the data path

   Async replication to the DR region for outages, plus point-in-time backups in a separate account for mistakes.

5. 5

   Walk through failover and failback

   Who decides, fencing the old primary, promoting the replica, switching DNS, and how the old primary rejoins.

6. 6

   Close with testing

   Scheduled restore tests and failover rehearsals, with measured RTO and RPO compared against the targets.