Orchestration vs Choreography: Who Runs Your Microservices?

When no service owns a job from start to end, a failed handoff can go unnoticed. Each service does its piece and passes the job on, and nobody checks that the next piece happened. Choosing who owns that job is what orchestration vs choreography is about.

Picture a common setup. A vet-at-home booking app lets dairy farmers book a vet for a sick buffalo, and pet owners book one for a dog with a fever. The setup is made up, but the problem is real.

One booking touches four parts of the system. The farmer pays a visit fee of Rs 600 (600 rupees), a vet is assigned, medicines are sent out, and the farmer gets an SMS. Each part is a separate service with its own database.

The team built it so each service reacts to the one before it. The booking service announces a new booking. The payment service hears that, holds the fee and announces that it did. The vet service hears that and assigns a vet.

For eight months it worked at about 2,500 bookings a day. Then support saw 140 complaints in one week. Farmers had paid, but no vet ever came.

The logs showed the cause. The payment service had held every fee and announced it. But the vet service had marked those messages as read before finishing its work, then crashed during a deploy. When it restarted, it skipped them.

Nothing in the system was waiting for the vet step, so nothing raised an alarm. The real question was who should know where each booking is. That question is the heart of orchestration vs choreography.

You need six words: microservice, command, event, message broker, saga and compensating step. Each one gets a plain meaning and an example from the booking app.

Microservice. A microservice is a small program that owns one job and its own data, and talks to other services over the network. Example: the payment service owns fees and refunds, and nothing else.

Command. A command is a message that tells one service to do something, such as hold Rs 600 for booking 8812. The sender expects a reply saying it worked or failed.

Event. An event is a message that announces something already happened, such as PaymentHeld. The sender does not know or care who listens to it.

Message broker. A message broker is a program that sits between services and passes messages along. In the publish-subscribe style used here, a service subscribes to a topic, a named stream of events, and gets every event published there. Kafka and RabbitMQ are common brokers.

Saga. A saga is a business job split into local steps, one per service, where each step saves its own data on its own. Booking a vet visit is a saga with four steps. The idea comes from a 1987 database paper by Hector Garcia-Molina and Kenneth Salem.

Compensating step. A compensating step is the undo for an earlier step that already succeeded. Example: if no vet is free, the compensating step releases the Rs 600 hold back to the farmer.

Orchestration and choreography are two ways to coordinate a job that spans several services. In orchestration, one central service directs every step. In choreography, each service reacts to events, and nobody directs the whole flow.

Think of the band at a Punjabi wedding procession. In one band, the band master walks in front with a whistle. He signals when the trumpets start, when the drums speed up, and when everyone stops at the bride's gate.

That is orchestration. The band master is the orchestrator, and his whistle signals are the commands. The players know only their own part and his signals, not the whole programme.

Now picture a group of dhol players with no leader. One starts a beat. The next hears it and adds a layer, and a third joins when he hears the second. The rhythm spreads from player to player.

That is choreography. Each beat a player hears is an event, and each player decides how to respond. Nobody waits for permission. But if one player misses his cue, nobody is in charge of noticing.

Where the picture breaks: musicians hear each other at once and never hear a beat twice. Services can receive an event late, twice or not at all, and the engineer sections deal with exactly that.

The booking app was built like the dhol players. Its stuck bookings were a missed cue that nobody noticed, because nobody was watching the whole procession.

In orchestration, one service called the orchestrator holds the steps of the job. It sends a command to each service in turn, waits for the reply, and then picks the next step.

The other services become simpler. The payment service does not know a vet comes next. It only knows how to hold a fee, take it for real, or release it when someone asks.

Here is a booking redrawn with an orchestrator. Look at the box in the middle: almost every arrow starts there. The services never call each other directly.

The progress table fixes the stuck-booking problem. For any booking, one row says which step it reached and what happened. Support can look it up in seconds instead of searching four sets of logs.

In choreography, each service listens for the events it cares about, does its own work, and publishes a new event. The flow comes from these reactions, and no service holds the whole plan.

This is how the booking app first worked. Follow the arrows below. Each arrow label is an event, and every event really travels through the message broker, which is left out to keep the picture clear.

The big win is loose coupling, which means services depend on each other as little as possible. The booking service does not even know the payment service exists. A new listener for a side effect, like an SMS, can join without changing anyone else.

The cost is visibility. Martin Fowler's 2017 article on event-driven systems says such a flow is hard to see because it is not explicit in any program text. Often, he writes, the only way to find it is by monitoring a live system.

That is what the team faced. To find where booking 8812 stopped, they had to search the logs of four services and line them up by time.

The main difference is where the flow lives. In orchestration it lives in one service you can read, test and watch. In choreography it is spread across many services, and each knows only its own reaction.

Choreography is not free of coupling, though. Services still depend on the shape of each event. If someone renames a field in PaymentHeld, the vet service breaks, just as it would with a changed command.

Orchestration has its own cost. The orchestrator is one more service to deploy, scale and keep alive. If it goes down and has no spare copy, no new booking moves at all.

Both styles undo a failed saga with compensating steps, but they differ in who triggers them. In orchestration, the orchestrator calls each undo. In choreography, each service hears a failure event and undoes its own work.

A database transaction is a group of changes that all happen or none do. If one fails, it rolls back, undoing every change in the group. A saga has no automatic rollback like that. You write every undo yourself. Sagas also lack isolation, so another request can see a half-finished booking, such as a fee held with no vet yet.

Here is the failure the booking app sees most: no vet is free near the village. Start at the question box and follow the No arrow to see the undo path.

The picture is the same in both styles. What changes is who walks the No path. These steps follow booking 8812 through each style.

1. 1

   Orchestration: the vet step fails

   The vet service replies that no vet is free for booking 8812. The orchestrator marks the step as failed in the progress table.

2. 2

   Orchestration: the orchestrator runs the undo

   It tells the payment service to release the hold, waits for the reply, then saves CANCELLED in the progress table. One function holds this whole path.

3. 3

   Choreography: the vet service announces

   The vet service publishes VetUnavailable. It does not know who is listening or what they will undo.

4. 4

   Choreography: each listener undoes its part

   The payment service hears it and releases the hold. The SMS service hears it and texts the farmer. If a listener is down, its undo waits until it comes back, as long as the broker keeps the event.

The orchestrator is waiting for a reply, so it can use a timeout. In the booking app it waits ten minutes, treats silence as a failure and starts the undo.

In pure choreography, silence produces no event, so nothing happens. That was the booking app's bug. The fix there is a separate checker that looks for bookings stuck in one state too long.

Choose orchestration when a job has several steps that must finish in order, with clear undo rules. Choose choreography when services just need to learn that something happened and can react on their own.

These signs point to orchestration:

• The job has three or more steps, and their order matters.
• Money is involved, so every failure needs a clear, tested undo.
• Support staff need to see where any single job is, right now.
• Steps need timeouts, like giving up on a vet reply after ten minutes.
• One team owns the whole job and can own the orchestrator too.

These signs point to choreography:

• Many services want to react to the same fact, like a confirmed booking.
• The reactions are independent, so one failing does not break the others.
• Different teams own the listeners and want to ship without asking each other.
• The flow is short, two or three hops, and easy to follow.

The choice is per flow, not per company. The booking app can orchestrate bookings and still choreograph everything that happens after a booking is confirmed.

Yes. A common pattern orchestrates the core job that must succeed, then publishes one event when it finishes, so other services can react on their own.

The team's new design does exactly this. The orchestrator owns the fee, vet and medicine steps. When the booking is confirmed, it publishes BookingConfirmed. The SMS service and the daily reports listen for that event, and the orchestrator never waits for them.

This split follows a simple test. If a step failing should cancel the booking, it belongs inside the orchestrator. If a step failing should just be retried later, like a late SMS, it can be a listener.

The split also keeps the orchestrator small. Business rules like vet pricing or medicine stock checks stay inside the services that own them. The orchestrator decides only the order, the timeouts and the undo.

The team moved bookings to the new design over three weeks. Old bookings finished on the old event chain, and new ones went through the orchestrator. After the switch, support could find a stuck booking with one lookup.

A saga orchestrator is a function that runs each step, remembers the undo for every step that succeeded, and runs those undos newest first when a later step fails.

The payments, vets and medicines objects below are stand-ins that send a command and wait for the reply. Each one raises StepFailed if its service says no.

def book_visit(booking):
    undo_steps = []
    try:
        payments.hold(booking.id, booking.fee)
        undo_steps.append(payments.release)
        save_progress(booking.id, "FEE_HELD")
        vets.assign(booking.id, booking.village)
        undo_steps.append(vets.unassign)
        save_progress(booking.id, "VET_ASSIGNED")
        medicines.send(booking.id, booking.medicines)
        save_progress(booking.id, "CONFIRMED")
    except StepFailed as err:
        for undo in reversed(undo_steps):
            undo(booking.id)
        save_progress(booking.id, "CANCELLED", reason=str(err))

Each undo_steps.append adds the undo for a step that just succeeded. If any later step fails, the loop runs those undo steps newest first. Each save_progress call writes the step to the progress table.

What just happened? We ran this function on Python 3.13 with fake services. When the vet step fails, the payment hold is released and the last row is CANCELLED. When the medicine step fails, the vet is unassigned first, then the hold is released.

The medicine step has no undo in the list because it is the last step. Nothing runs after it, so nothing can fail after it and force it to be undone.

Where this sketch breaks

The undo list lives only in memory. If the orchestrator process crashes after the vet step, a restarted copy finds VET_ASSIGNED in the table but has no list of undos and no running function.

There is a smaller gap too. A crash between payments.hold and its save_progress call leaves a held fee with no row at all. A production orchestrator needs saved records it can rebuild from, and undo steps that are safe to run twice. If an undo itself fails, the loop stops and the hold is never released. A real orchestrator retries each undo until it succeeds, and alerts if it cannot.

Workflow engines save every step of a run outside the orchestrator process, so a crashed run can be rebuilt and continued. You write the steps as code or definitions, and the engine does the saving.

Temporal: replay from event history

Temporal is an open-source platform for durable workflows. Its docs describe an Event History for each run: a complete, durable log of everything that has happened in that run. After a worker, the process that runs your workflow and step code, crashes, Temporal replays that log to rebuild the run's state.

Replay means the engine runs your workflow code again from the beginning. Each step that already finished returns its recorded result instead of running again. So the payment service is not asked again to hold a fee it already held. A step that was still running at the crash may run again, so Temporal's docs recommend that such steps be idempotent, meaning safe to repeat.

This puts a rule on your code. The docs say a workflow must make the same decisions when given the same history. So it must not read values that differ between runs, such as the system clock or a random number.

The Python SDK gives safe versions instead. In temporalio 1.34.0, the latest release as of 19 October 2026, workflow.now() returns workflow time and workflow.random() returns a deterministically seeded generator.

Timeouts are built in too. An activity is Temporal's name for one step, a normal function that does one well-defined action. The same SDK version refuses to schedule an activity unless it has a start_to_close_timeout (time allowed for one attempt) or a schedule_to_close_timeout (total time, including retries). It raises a ValueError saying so.

Conductor: workflows as JSON definitions

Conductor is another open-source workflow engine, originally created by Netflix Engineering. As of 19 October 2026, its site describes Conductor OSS as the continuation of the original Netflix repository, under the Apache 2.0 licence.

In Conductor, the workflow is a JSON definition. Workers in your own services, plus Conductor's built-in tasks, run the steps. Its docs say it persists workflow and task state and supports recovery after worker and infrastructure failure.

You do not need an engine for one short flow. An ordinary service with a progress table works. An engine earns its place once you have several long flows, or flows that wait minutes or days.

You make a handler safe by recording the id of every event it has handled, in the same database transaction as its work. Then a second copy of the same event is skipped.

Here is part of the booking flow as event handlers. In real life each handler lives in its own service, and they sit together here only for reading. The subscribe and publish helpers are stand-ins for a broker's client library.

@subscribe("BookingRequested")          # runs in the payment service
def on_booking_requested(event):
    hold_payment(event["booking_id"], event["fee"])
    publish("PaymentHeld", {"booking_id": event["booking_id"]})

@subscribe("PaymentHeld")               # runs in the vet service
def on_payment_held(event):
    vet = find_nearest_vet(event["booking_id"])
    if vet is None:
        publish("VetUnavailable", {"booking_id": event["booking_id"]})
    else:
        publish("VetAssigned", {"booking_id": event["booking_id"], "vet_id": vet.id})

@subscribe("VetUnavailable")            # runs in the payment service
def on_vet_unavailable(event):
    release_payment(event["booking_id"])

Look at the last handler. When no vet is free, the vet service only announces it. The payment service has to listen for VetUnavailable and release the hold itself. The undo logic is spread across services, one handler at a time.

Why duplicates happen

microservices.io recommends a broker that guarantees at-least-once delivery. That means every event is delivered at least once, so the same event can reach a handler more than once.

Twice. We ran it on a fake broker, and hold_payment was called 2 times for one booking. The fix is called the idempotent consumer pattern. Idempotent means running it twice has the same effect as running it once. This version saves the event id and the hold in one transaction:

import sqlite3
db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE holds (booking_id INTEGER, amount INTEGER)")
db.execute("CREATE TABLE processed (event_id TEXT PRIMARY KEY)")

def on_booking_requested(event):
    try:
        with db:  # one transaction: both rows are saved, or neither
            db.execute("INSERT INTO processed VALUES (?)", (event["event_id"],))
            db.execute("INSERT INTO holds VALUES (?, ?)",
                       (event["booking_id"], event["fee"]))
    except sqlite3.IntegrityError:
        pass  # this event id was handled before, so skip it

event = {"event_id": "e-101", "booking_id": 8812, "fee": 600}
on_booking_requested(event)
on_booking_requested(event)  # the broker delivers the same event again
print(db.execute("SELECT COUNT(*) FROM holds").fetchone()[0])

Check it yourself: save it as a file and run it with Python 3. It should print 1, meaning one hold row. Remove the line that inserts into processed, and it prints 2.

The trick works only when the hold lives in the same database as the processed table. If the hold calls an outside payment gateway, pass the event id to the gateway as an idempotency key instead. That is a unique id the gateway uses to ignore repeats of the same request.

The other half: publishing reliably

The sending side has a matching problem. A service must save its data and publish its event together, or a crash between the two loses the event. A broker and a database cannot share one ordinary transaction.

The transactional outbox pattern solves this. The service writes the event into an outbox table in the same transaction as its data. A separate process then sends outbox rows to the broker. That relay may send a message more than once, which is why consumers must be idempotent.

You find stuck jobs with a checker that looks for jobs sitting in one step longer than they should. It is the missing owner for a choreographed flow, and a safety net for an orchestrated one.

Each service records the step a booking reached and when. The checker reads those rows on a schedule and alerts on anything unfinished and older than the limit. This sample uses a ten-minute limit:

import sqlite3
from datetime import datetime, timedelta

db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE progress (booking_id INTEGER, step TEXT, updated_at TEXT)")
now = datetime(2026, 10, 19, 10, 0)
rows = [
    (8811, "CONFIRMED", now - timedelta(minutes=50)),
    (8812, "FEE_HELD", now - timedelta(minutes=42)),
    (8813, "FEE_HELD", now - timedelta(minutes=3)),
]
db.executemany("INSERT INTO progress VALUES (?, ?, ?)",
               [(b, s, t.isoformat()) for b, s, t in rows])
cutoff = (now - timedelta(minutes=10)).isoformat()
stuck = db.execute(
    "SELECT booking_id, step FROM progress "
    "WHERE step NOT IN ('CONFIRMED', 'CANCELLED') AND updated_at < ?",
    (cutoff,)).fetchall()
print(stuck)

Check it yourself: run it with Python 3 and it prints [(8812, 'FEE_HELD')]. Booking 8811 is finished. Booking 8813 is unfinished but only 3 minutes old, so it is not stuck yet.

Store timestamps in UTC and in one format. The comparison here is plain text, which only sorts correctly when every value uses the same ISO 8601 layout.

To see one booking's whole path, put its id in every command, event and log line. Distributed tracing builds on that idea: each request gets one id, every service logs it, and a central service records when each operation started and ended.

The two classic mistakes are making the orchestrator too smart, and letting choreographed events grow into a tangle nobody can trace. Both are easier to fix early than late.

• A god orchestrator. Business rules creep in until it knows every detail of every service. Keep it to order, timeouts and undo.
• One fragile orchestrator. Run more than one copy and keep its state in a database, so any copy can continue a run. Let only one copy own each run at a time.
• Event chains nobody can draw. Ten services reacting to each other hide the real flow. Write the chain down and keep it current.
• Event loops. Service A reacts to B, and B reacts to A, so one event sets off an endless loop. Review every new listener for loops.
• Handlers that are not idempotent. A redelivered event charges or books twice. Record handled event ids in the same transaction as the work.

Explain it as a choice about where the flow lives, then pick a style per flow, with a reason tied to how failures are handled.

1. 1

   Name the job and its steps

   Say the job out loud: hold fee, assign vet, send medicines, notify. Mark which steps move money or need an undo.

2. 2

   Define both styles in one line each

   Orchestration means one coordinator sends commands. Choreography means services react to events through a message broker.

3. 3

   Pick a style for the core

   For a flow with money and undo steps, choose orchestration. Say why: one place to see state, run the undo and enforce timeouts.

4. 4

   Choreograph the side effects

   Publish one event when the core job finishes. Let SMS, reports and loyalty listen without slowing the main path.

5. 5

   Cover failure and crashes

   Explain compensating steps, idempotent handlers, the transactional outbox, and how the orchestrator survives a crash by saving its state.

6. 6

   Show how you would debug it

   Name the booking id in every message, the progress table, and an alert for jobs stuck too long.