Railway's September 30, 2026 Outage Explained: When a Deploy Beat Its Schema

Picture a boring change. A column changes type, the service that reads it gets a small update, staging passes, you merge. Six minutes later every domain on the platform returns 404 to new visitors.

That is roughly what happened at Railway on September 30, 2026. The outage lasted about five minutes, and Railway published a clear, honest report the same day. This piece walks through that report in plain English, then pulls out the habits that stop the same bug in your systems. It is a learning piece, not a dunk: most teams that run migrations have this race hiding somewhere.

For roughly five minutes, between about 07:35 and 07:40 UTC, new connections to domains hosted on Railway returned HTTP 404 Not Found in all regions. That covered both custom domains and the default *.up.railway.app domains, for all customers.

The impact was narrower than "Railway was down" suggests. Railway reports that existing connections were not disrupted, no data was lost, and user services kept running. What broke was access. One easy-to-miss casualty: the routing service also resolves private network DNS, so private networking lookups between services failed in the same window.

The timeline (all times UTC, September 30, 2026)

Notice that recovery happened before diagnosis, with no rollback. The migration finished, the database matched what the new binary expected, and lookups started succeeding again.

The root cause was an ordering bug: the new routing service came online before the database schema it depended on existed. A race in the pipeline configuration ran the schema update in parallel with the deployment, and the deployment won.

The change itself was ordinary: a modified data type in the database that stores routing data, plus a new routing service version that reads the new table format. Schema first, then the code that reads it.

So why did the order break? Railway explains that the routing service used to live inside a larger monolith, the non-distributed control plane, where a single pipeline enforced the order. They split it out into a new distributed control plane for resiliency, which parallelized deployments. Now the migration job runs in one system and the routing service is built by another.

Splitting a monolith rarely breaks the code. It breaks implicit guarantees the monolith gave you for free, like "migrations run before the new binary starts", which nobody wrote down because the single pipeline made them true by accident.

Why did staging not catch it?

Railway's report says the change was tested in staging but does not say why staging missed the race. My inference, not Railway's statement: a common way this happens is that the staging migration is small and finishes before the service restarts. Races depend on timing, and a passing test proves the code works, not that the rollout order is guaranteed.

The 404 came from Railway's origin proxies, not from your app and not from the routing service. For every request, an origin proxy asks the routing service which service the domain should go to. During the incident the lookup itself failed, and the proxies handled that failure the same way they handle a domain with no routes: by returning 404.

There is a second detail that turned a lookup failure into platform-wide impact. Railway says the change in question was a change to the domain fallback behavior. Because that fallback had been modified, the proxies could not fall back to the last route they had seen for a domain. A stale-but-correct route would have kept most traffic flowing. Without it, every new connection depended on a live lookup that was failing.

A five-minute race hits every region when the rollout is global and simultaneous. Railway's report says the new routing service rolled out within every region, so every region ran the new binary against the old schema at once, with none left on the old version.

Regions protect you from hardware and data centre failures. They do not protect you from your own deploy pipeline, because the pipeline is the one thing every region shares.

The report does not say whether the routing data store is global or per region, so I will not guess. It does not matter much: what was global was the rollout, plus the fallback change that removed each proxy's cushion.

Railway lists three fixes, and they map neatly onto the three things that went wrong: ordering, verification and blast radius.

1. 1

   Forcing serial ordering of the migration rollout

   Unified deployment pipelines so the migration and binary cannot race, plus dry runs of migrations.

2. 2

   Blocking the binary's blue/green rollout on schema validation

   Rollouts independently validate the schema version they were authored with, and the service refuses to go live until the matching database schema is in place.

3. 3

   Shard-based rollouts per region

   Railway says it will be adding shard-based rollouts to the new distributed control plane to isolate impact. This is stated as upcoming work.

The second fix is the interesting one. Serial ordering fixes this pipeline. Schema validation inside the service also protects every future pipeline, including the one someone misconfigures next year, because the service checks for itself.

Run the schema change first, verify it landed, then deploy the code that needs it, and make the pipeline enforce that order rather than hoping timing works out. If old code and new code will both be running during the rollout, which they will, the schema must work for both at the same time.

That second sentence is the part people skip. During any rolling or blue/green deploy, old and new instances serve traffic together for a while. A schema only the old code understands breaks the new instances, which is what happened here. The way out is a migration style where every intermediate state works for both versions.

An expand and contract migration (also called parallel change) is a way to change a schema in small steps where every step is compatible with both the old and the new code. You expand the schema to support both shapes, migrate readers and data over, then contract by removing the old shape once nothing uses it.

Danilo Sato's write-up on Martin Fowler's site describes the three phases and the useful property: you can safely release after any phase. If a deploy races ahead or you roll back, nothing breaks, because no step removes something a running version still reads.

Here is a hypothetical example in PostgreSQL. Say your routes table stores the upstream as plain text, and you want to change it to a structured JSONB value so one domain can point at several targets. The tempting version is a single ALTER COLUMN TYPE plus a code deploy. The safe version looks like this.

-- Phase 1: EXPAND (additive, old code keeps working)
ALTER TABLE routes ADD COLUMN upstreams JSONB NULL;

-- Phase 2: MIGRATE (backfill in small batches to avoid long locks)
UPDATE routes
SET    upstreams = jsonb_build_array(jsonb_build_object('host', upstream))
WHERE  upstreams IS NULL
  AND  id IN (SELECT id FROM routes WHERE upstreams IS NULL LIMIT 5000);
-- repeat until 0 rows updated

-- Deploy code that WRITES both columns and READS upstreams,
-- falling back to upstream when upstreams is NULL.

-- Phase 3: CONTRACT (a later release, after every reader is on the new column)
ALTER TABLE routes ALTER COLUMN upstreams SET NOT NULL;
ALTER TABLE routes DROP COLUMN upstream;

The trade-off is time and discipline. One change becomes three releases, you carry dual-write code for a while, and someone must remember the contract step. In exchange, deploy order stops being a single point of failure. For hot-path tables like routing, auth or payments, that is worth it. For an admin table nobody reads during the deploy, a direct change can be fine.

A readiness check that verifies the service's real dependencies, including the schema version, stops a new instance from taking traffic until it can actually do its job. This is what Railway's second fix does: the service refuses to go live until the matching schema is in place.

In Kubernetes, a failing readiness probe removes the pod from the Service's endpoints without restarting it. Most teams wire readiness to "the HTTP server is listening", which proves little. A better check asks: can I reach the database, is the schema the version I was built for, and is my required config loaded?

// Built into the release: the schema version this code was written against.
const REQUIRED_SCHEMA_VERSION = 42;

let ready = false;

async function checkSchema(db: { query: (sql: string) => Promise<{ version: number }[]> }) {
  const [row] = await db.query('SELECT max(version) AS version FROM schema_migrations');
  if (!row || row.version < REQUIRED_SCHEMA_VERSION) {
    ready = false; // stay out of rotation, keep the old version serving
    return;
  }
  ready = true;
}

// GET /readyz
export function readyz(): { status: number; body: string } {
  return ready
    ? { status: 200, body: 'ok' }
    : { status: 503, body: 'waiting for schema ' + REQUIRED_SCHEMA_VERSION };
}

Two cautions. A readiness gate only helps if the old version keeps serving while the new one waits; if the rollout replaces every instance at once, a perfect check just means zero ready instances. And keep checks cheap and scoped, or one slow dependency can mark your whole fleet unready.

When the lookup fails, as opposed to returning "no such domain", the safest choice is usually to serve the last known good route, and if there is none, return 503 with a Retry-After header. Returning 404 tells the client something false: that the resource does not exist.

This matters beyond wording. RFC 9110 says a 404 is heuristically cacheable, so caches may store it, while 503 signals a temporary problem and may carry Retry-After. Crawlers and client code may also read 404 as "gone". The fix is to keep "not found" and "lookup failed" apart in the type system.

type RouteLookup =
  | { kind: 'found'; target: string }
  | { kind: 'not_found' }               // the domain truly has no route
  | { kind: 'error'; cause: unknown };  // we could not find out

function respond(lookup: RouteLookup, lastKnown?: string) {
  switch (lookup.kind) {
    case 'found':
      return { proxyTo: lookup.target };
    case 'not_found':
      return { status: 404 };
    case 'error':
      // Fail static: keep serving the last good answer if we have one.
      if (lastKnown) return { proxyTo: lastKnown };
      return { status: 503, headers: { 'Retry-After': '5' } };
  }
}

For a service starting up without its schema or config, fail closed: refuse to become ready. For a router in the hot path, failing static (continuing with the last known good config) is usually better for users. The trade-off is that a stale route can point at a moved or deleted service, so cap how long a cached route may be reused.

A canary or regional rollout limits the blast radius by sending a new version to a small slice of traffic first, watching error rates, and only then promoting it further. If Railway's routing service had reached one shard first, the race would most likely have broken that shard for a few minutes instead of every region. That is my inference, and it is exactly the reasoning behind their third fix.

• Start with one shard or region that carries real but limited traffic.
• Define the promotion signal before the rollout: 5xx rate, 404 rate on known-good domains, lookup error rate, latency.
• Hold each wave long enough to cover the slowest dependent step. A canary that is promoted in 60 seconds cannot catch a migration that takes five minutes.
• Automate the halt. A rule that stops promotion when error rate jumps beats a human watching a dashboard.

The cost is speed: a global push takes minutes, careful waves can take hours. A common compromise is fast waves for low-risk services and slow waves for anything in front of all traffic, like routing, DNS, auth and config. You would not push a new UPI payment routing rule to every transaction during a festival sale either.

Declare the dependency explicitly in the pipeline so the deploy job cannot start until the migration job has succeeded. In GitHub Actions, jobs run in parallel unless you link them with needs, and if a needed job fails, the jobs that depend on it are skipped by default.

jobs:
  migrate:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v4
      - run: ./scripts/migrate --dry-run
      - run: ./scripts/migrate --apply
      - run: ./scripts/assert-schema-version "$(cat db/SCHEMA_VERSION)"

  deploy-canary:
    needs: migrate            # cannot start until migrate succeeds
    runs-on: ubuntu-latest
    steps:
      - run: ./scripts/deploy --shard canary
      - run: ./scripts/watch-metrics --minutes 15 --max-error-rate 0.5

  deploy-all:
    needs: deploy-canary      # only after the canary held
    runs-on: ubuntu-latest
    steps:
      - run: ./scripts/deploy --regions all --waves 3

The scripts are placeholders. The shape is what matters: explicit edges and no path where deploy starts without migrate. Railway's case was harder because the migration and the binary lived in two separate systems, where a needs keyword cannot help and the service-side schema check becomes the real guarantee.

Questions like "how would you change a column type on a table that serves every request?" show up in system design and SRE rounds. The interviewer wants to hear that you think about the window where two versions coexist.

A strong answer also mentions rollback: if you cannot roll back the code without rolling back the schema, the migration was not backward compatible.

• Is every step of this migration compatible with both the currently running code and the new code?
• Does the pipeline make the deploy wait for the migration, with an explicit dependency rather than timing?
• Does the service check the schema version it was built for and refuse to become ready if it does not match?
• Does the code distinguish "not found" from "could not look it up", and return something other than 404 for the second case?
• Is there a last known good fallback for hot-path config, and is this release changing it? If so, can that change ship separately?
• Does the rollout reach one shard or region first, with a metric that halts it automatically?
• Is the contract step (dropping the old column) scheduled as its own later release?

If you take one thing from this incident, make it the readiness check. Pipelines get rewritten and monoliths get split, but a service that checks its own schema keeps protecting you. Pick one service you own and see what its readiness probe actually verifies.