Event-Driven Architecture with Fan-Out Queues

Checkout becomes as slow as the slowest service it calls, and one failure can leave an order half done. This happens when the store calls each service in turn while the customer waits. Event driven architecture removes that wait.

Picture a common setup. An online store sells handmade quilts. When a customer pays, billing writes a GST (goods and services tax) invoice, packing prints a warehouse slip, and an SMS confirms the order. The setup is made up, but the problem is real.

The first version did all three inside the checkout request. Follow the arrows from the customer. Notice that checkout must wait for every box on its right before it can reply.

During a Diwali sale, the SMS provider slowed down for 40 minutes. Each SMS call took 25 seconds instead of half a second (both made up). Checkout requests piled up and timed out, and customers gave up.

Worse, some orders were left half done. The invoice was written, but the packing call never ran. The warehouse found the gaps only when customers phoned to ask where their quilt was.

The fix starts from one observation. Sending the SMS is not part of taking the order, so the order should not wait for it. The rest of this guide rebuilds the flow with events and fan-out queues.

You need six words. Each comes with its example from the quilt store.

• Event: a record of something that already happened, so its name is in the past tense. Example: OrderPlaced, with the order id, the items and the amount.
• Producer: the service that publishes an event. Example: checkout publishes OrderPlaced after it saves the order.
• Consumer: a service that reads events and does its own work with them. Example: packing reads OrderPlaced and prepares a slip.
• Topic: a named channel that producers publish to. It keeps a list of subscribers and pushes a copy of every event to each one. Amazon SNS, the Simple Notification Service, provides topics.
• Queue: a waiting line of messages that a consumer pulls from at its own pace. A message stays until the consumer deletes it or the retention period ends. Amazon SQS, the Simple Queue Service, provides queues.
• Subscription: the link between one topic and one receiver, such as a queue. Example: the SMS queue's subscription asks the order-events topic for a copy of each event.

Event driven architecture is a design where services tell each other what happened, through events, instead of telling each other what to do, through direct calls. The producer neither knows its consumers nor waits for them.

In the old style, checkout calls billing and waits for the invoice. In the new style, checkout announces that order 4417 was placed, then moves on.

The big win is loose coupling, which means services depend on each other as little as possible. Checkout no longer needs to know that an SMS service exists. Adding a fourth consumer later needs no change to checkout.

Martin Fowler, a well-known software author, wrote in 2017 that "event-driven" covers four different patterns. Two of them decide what goes inside an event. The other two are about how you store and read data. Event sourcing records every change as an event, and CQRS uses separate models for writing and for reading.

What an event should carry

Event notification sends a small event that says what happened, with an id. Consumers call the producer back if they need more. Events stay tiny, but every consumer that calls back now depends on the producer staying up.

Event-carried state transfer puts the data consumers need inside the event, such as the address and the items. Fowler lists the gains as resilience and lower latency, because no remote call is needed. The cost is lots of copies of the same data.

The store chose the second style for OrderPlaced. Packing needs the address and quilt sizes, and SMS needs the phone number. Asking checkout for these on every event would bring back the old coupling. The event has these fields.

• event_id: a unique id for this one event, used later to spot duplicates.
• event_type: OrderPlaced, so filters and consumers know what happened.
• order_id: 4417, the business id that ties every log line together.
• occurred_at and version: when it happened, and which version of the order this is.
• data: the items, the amount, the delivery address and the phone number.

In SNS and SQS fan-out, the producer publishes one event to an SNS topic, and SNS puts a copy into every SQS queue subscribed to it. Each consumer then reads only its own queue.

SNS pushes each message out to many subscribers. SQS holds each copy until its own consumer is ready. In the diagram below, notice that checkout has only one arrow.

Now replay the Diwali problem. The SMS provider slows down again, so the SMS worker slows down and messages build up in its queue. Checkout replies as soon as the order is saved and published, and billing and packing do not notice.

1. 1

   Save the order

   Checkout writes order 4417 to the orders database. Nothing is announced until that write succeeds.

2. 2

   Publish once

   Checkout publishes OrderPlaced to the order-events topic and shows the customer the confirmation page.

3. 3

   SNS makes the copies

   SNS sends a copy to each subscribed queue whose filter matches. From here, each copy lives its own life.

4. 4

   Workers pull at their own pace

   Each worker receives from its own queue when it is ready, does its job, then deletes the message.

5. 5

   Slow parts only slow themselves

   If the SMS worker is down, its copies wait in the SMS queue. Billing and packing carry on.

Two setup details that trip up beginners

First, each SQS queue needs an access policy that lets the topic call sqs:SendMessage on it. If the policy blocks SNS, AWS counts that as a client-side error. SNS does not retry it, and drops the message unless the subscription has its own dead-letter queue.

Second, SNS normally wraps each message in a JSON envelope with fields such as Type, MessageId, TopicArn and Message. Your event sits inside Message as a string. Raw message delivery strips the envelope, so the queue receives your event exactly as published.

Raw delivery has one limit to know. As of 19 October 2026, AWS documents that an SQS subscription with raw delivery takes at most 10 message attributes. SNS discards a message that carries more.

Crashing between saving and publishing

If checkout crashes after saving the order but before publishing, the order exists but no event goes out, and nobody packs the quilt. This is the dual-write problem: a database and a topic are two separate systems, and one step cannot update both.

The usual fix is the transactional outbox, described by Chris Richardson on microservices.io. Checkout writes the order and an outbox row in the same database transaction, so both happen or neither does. A separate relay process reads new outbox rows and publishes them.

The pattern page states the catch plainly: the relay might publish a message more than once. So every consumer must handle duplicates, which a later section shows how to do.

Fan-out gives every subscriber its own copy of each message, while a work queue gives each message to just one of several competing workers. Use fan-out for different jobs, and a work queue for sharing one job.

Think of a city fire brigade. When an alarm reaches the control room, the operator relays it to every fire station that must know. Each station writes it into its own logbook and acts on it separately. That is fan-out.

Inside one fire station, the crews share a single duty roster. The next call goes to whichever free crew is at the top, not to every crew. That is a work queue: one copy, taken by one worker.

Where the picture breaks: a crew never gets the same call twice by accident. A queue can hand the same message over twice, so your code must do the remembering.

The old debate of pub-sub versus message queues is really this choice. Pub-sub, short for publish-subscribe, means fan-out to subscribers. A message queue used on its own usually means a work queue.

SNS plus SQS uses both at once. The topic fans out to queues, and each queue is a work queue for its own workers. In the Diwali rush, the store ran three packing workers on the packing queue.

You add a subscription filter policy, a rule on the subscription that says which events it accepts. SNS checks each event against the policy and does not send non-matching events to that subscriber.

The default is the opposite. AWS states that a subscription without a filter policy receives every message published to its topic.

The store's topic now carries OrderPlaced, OrderShipped, OrderCancelled and RefundIssued. The SMS queue needs only placed and shipped events. Packing needs a filter too, or it will receive every refund.

A filter can match message attributes, small key and value labels sent beside the message. It can also match fields inside the message body. The filter policy scope picks which one. The store sets an event_type attribute on every publish.

Here is the SMS side as a complete CloudFormation template, a YAML file that describes cloud resources. It creates the topic, the SMS queue, its dead-letter queue, the access policy and the filtered subscription.

Resources:
  OrderEventsTopic:
    Type: AWS::SNS::Topic

  SmsDeadLetterQueue:
    Type: AWS::SQS::Queue
    Properties:
      MessageRetentionPeriod: 1209600

  SmsQueue:
    Type: AWS::SQS::Queue
    Properties:
      VisibilityTimeout: 60
      RedrivePolicy:
        deadLetterTargetArn: !GetAtt SmsDeadLetterQueue.Arn
        maxReceiveCount: 5

  SmsQueuePolicy:
    Type: AWS::SQS::QueuePolicy
    Properties:
      Queues: [!Ref SmsQueue]
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal: {Service: sns.amazonaws.com}
            Action: sqs:SendMessage
            Resource: !GetAtt SmsQueue.Arn
            Condition:
              ArnEquals: {aws:SourceArn: !Ref OrderEventsTopic}

  SmsSubscription:
    Type: AWS::SNS::Subscription
    Properties:
      TopicArn: !Ref OrderEventsTopic
      Protocol: sqs
      Endpoint: !GetAtt SmsQueue.Arn
      RawMessageDelivery: true
      FilterPolicyScope: MessageAttributes
      FilterPolicy:
        event_type: [OrderPlaced, OrderShipped]

Four settings do the work. FilterPolicy lets two event types in. RawMessageDelivery removes the envelope. RedrivePolicy moves a message to the dead-letter queue after five receives. The queue policy lets only this topic send to the queue.

To check the template before deploying, run cfn-lint, the CloudFormation linter, on it. Version 1.57.2 printed nothing and exited with code 0 on this file. Change VisibilityTimeout to 50000 and it reports E3034, because the maximum is 43200 seconds (12 hours).

We also ran the same fan-out on moto 5.2.4, a local AWS emulator. Publishing OrderPlaced, RefundIssued and OrderShipped put 3 messages in an unfiltered queue and 2 in the SMS queue. An emulator is not AWS, so treat that as a check of your logic only.

The message is not lost straight away. It reappears in the queue after the visibility timeout and is tried again. With a dead-letter queue set up, SQS moves it there after too many failed receives. Without one, it keeps coming back until the queue's retention period ends, 4 days by default, and SQS deletes it.

When a worker receives a message, SQS hides it from other workers for the visibility timeout. The default is 30 seconds and the maximum is 12 hours, counted from the first receive. The worker must delete the message before the timer ends.

If the worker crashes or hits an error, it never deletes the message. The timer runs out, the message becomes visible again, and the same worker or another one picks it up.

Some messages fail every time, such as an order whose phone number is missing a digit. A dead-letter queue, or DLQ, is a side queue that catches these. The redrive policy's maxReceiveCount sets how many receives a message gets first.

Follow one SMS message through the diagram below. Notice the two ways out: a delete when it works, and the dead-letter queue when it keeps failing.

In our moto test with a 1-second timeout and maxReceiveCount 5, the message was received 5 times without a delete. The sixth receive found the source queue empty, and the message sat in the DLQ.

Pick the visibility timeout from your real processing time. AWS suggests starting at the longest time a message normally takes. Too short, and a healthy worker loses its message to a second worker. Too long, and a crashed worker's message waits ages for its retry.

For jobs of uneven length, the worker can extend its own timer with ChangeMessageVisibility while it is still busy. AWS calls this a heartbeat. The 12-hour limit still applies.

Three details the AWS docs mention in passing

First, for standard queues with maxReceiveCount above 3, SQS moves a message received 3 or more times without deletion to the back of the queue. So a failing message can wait behind newer ones before it reaches the DLQ.

Second, in standard queues a message moved to the DLQ keeps its original enqueue timestamp. If the DLQ keeps messages for less time than the source queue, old failures can expire before anyone looks. AWS advises a longer retention period on the DLQ, so the template sets 14 days, the maximum.

Third, the DLQ alarm needs the right metric. NumberOfMessagesSent does not count messages that SQS moved there after maxReceiveCount. Alarm on ApproximateNumberOfMessagesVisible above 0 instead. These three are documented as of 19 October 2026.

To look at a queue by hand, ask SQS for two counters. Replace QUEUE_URL with your queue's URL.

aws sqs get-queue-attributes --queue-url QUEUE_URL \
    --attribute-names ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible

The output looks like this.

{
    "Attributes": {
        "ApproximateNumberOfMessages": "2",
        "ApproximateNumberOfMessagesNotVisible": "1"
    }
}

ApproximateNumberOfMessages counts messages waiting, and ApproximateNumberOfMessagesNotVisible counts messages received but not yet deleted. Both are message counts, and AWS calls them approximate. Here, 2 messages wait and 1 is in flight. AWS describes ApproximateNumberOfMessages and the CloudWatch metric ApproximateNumberOfMessagesVisible the same way: messages available for retrieval.

Once the bug is fixed, a DLQ redrive moves the messages back to the source queue for another try.

When SNS cannot reach the queue

SNS retries delivery by itself. For SQS and AWS Lambda endpoints (Lambda runs your function on demand), the default policy makes 100,015 attempts over 23 days. For SMS, email and mobile push endpoints, it makes 50 attempts over 6 hours. Only HTTP/S subscriptions let you change the policy.

When the retries run out, or after a client-side error such as a blocking queue policy, SNS discards the message. The exception is a subscription with its own dead-letter queue. That is a different DLQ from the one on the SQS queue.

You make every consumer idempotent, so it gives the same result whether it sees an event once or five times. You also carry a version in each event, so older events can be ignored.

The SQS documentation is direct about this. Standard queues deliver every message at least once, but more than one copy might be delivered. Messages may also occasionally arrive out of order.

The visibility timeout does not prevent this either. AWS states there is no absolute guarantee that a message will not be delivered twice during that window.

At the store, a duplicate OrderPlaced once printed two packing slips (made up). Two identical quilts went to one customer, and the courier charged for both.

Pause and predict: the worker first asks "have I seen this event_id?", prints the slip, then records the id. What happens if it crashes right after printing?

The id was never recorded, so the retry prints a second slip. The check and the work must be saved together, in one transaction.

The worker below does that. It receives up to 10 messages, waiting up to 20 seconds, the long-polling maximum. It saves the event id and the slip row in one SQLite transaction, and a separate printer job prints from the slips table.

import json
import os
import sqlite3
import boto3

sqs = boto3.client("sqs")
QUEUE_URL = os.environ["PACKING_QUEUE_URL"]
db = sqlite3.connect("packing.db")
db.execute("CREATE TABLE IF NOT EXISTS processed (event_id TEXT PRIMARY KEY)")
db.execute("CREATE TABLE IF NOT EXISTS slips (order_id INTEGER, event_id TEXT)")

def handle(event):
    try:
        with db:  # one transaction: both rows are saved, or neither
            db.execute("INSERT INTO processed VALUES (?)", (event["event_id"],))
            db.execute("INSERT INTO slips VALUES (?, ?)",
                       (event["order_id"], event["event_id"]))
    except sqlite3.IntegrityError:
        pass  # this event_id was saved before, so it is a duplicate

def poll_once():
    resp = sqs.receive_message(QueueUrl=QUEUE_URL, MaxNumberOfMessages=10,
                               WaitTimeSeconds=20)
    for msg in resp.get("Messages", []):
        handle(json.loads(msg["Body"]))  # raw delivery: the body is the event
        sqs.delete_message(QueueUrl=QUEUE_URL,
                           ReceiptHandle=msg["ReceiptHandle"])

The event_id column is a primary key, so a second insert of the same id fails with IntegrityError. The with block rolls back the whole transaction, so the slip row is never saved twice.

We ran this exact code against moto with order 4417 sent twice and order 4418 once. It saved 2 slip rows, not 3, and left the queue empty. When the second insert failed for another reason, the event id was not saved either, so a retry would process it.

The delete runs either way. A duplicate that is never deleted would come back after each timeout until a DLQ or the retention period removes it.

SQLite keeps the example runnable. In production, every worker on the queue must use one shared database for the processed table, or a duplicate that reaches two workers prints twice. The printer job has the same gap one step later, so it should mark each slip printed in the same way.

Order matters for some events. If OrderCancelled arrives before OrderPlaced, a careless packing worker would print the slip anyway. Store the latest version seen for each order, and ignore any event older than that.

When strict order truly matters, the whole path must be FIFO (first in, first out): an SNS FIFO topic with SQS FIFO queues subscribed. AWS documents that this pairing delivers in publish order without duplicates.

Messages with the same message group ID are processed one at a time, in order. Use the order id as the group, so each order stays in sequence while different orders run in parallel.

In our moto test, a worker holding the first event of order 4417 blocked that order's second event, while order 4418 kept flowing. A worker that crashes before deleting still gets its message again, so keep the idempotency check.

Skip events when the caller needs the answer right now, or when one simple service does the whole job. Events add moving parts, and they pay off when several independent reactions follow one change.

• The user must see the result before moving on, like whether a payment went through. That needs a direct call with a clear yes or no.
• There is only one consumer, and a second one is unlikely. A direct call or a plain work queue is simpler.
• Your team cannot yet watch queue depth, dead-letter queues and consumer delay. Events without monitoring fail silently.
• Several steps must all succeed or all be undone together. That calls for a saga, a chain of steps where each finished step has an undo step that runs if a later one fails.

Fowler's warning about event notification is about visibility. The full business flow is no longer written in one place in the code. He warns that the flow then gets hard to see, and often shows up only when you monitor the live system.

The store's answer is boring and effective. Every event carries a correlation id, here the order id, and every worker logs it. One search for order 4417 shows its whole journey across all three queues. The store also watches these numbers.

• ApproximateNumberOfMessagesVisible on each queue, the number of messages waiting, with an alarm when the SMS queue passes 500.
• ApproximateAgeOfOldestMessage, in seconds, which reveals a stuck consumer even when the queue looks short.
• ApproximateNumberOfMessagesVisible on each DLQ, which should be 0 on a normal day.
• Time from OrderPlaced to the SMS being sent, a metric the store records itself for every order.

Start from a direct-call design that fails, then add one event and a fan-out, and finish with failure handling. Interviewers want to hear the trade-offs, not just see the boxes.

1. 1

   Show the coupled version

   Draw checkout calling three services in a row. Point out that the slowest one sets checkout's speed, and one failure leaves orders half done.

2. 2

   Publish one event

   Checkout saves the order and publishes OrderPlaced. Mention an outbox, so a crash between the two cannot lose the event.

3. 3

   Fan out to queues

   Draw one topic and one queue per consumer. Say why: each consumer gets its own pace, its own retries and its own scaling.

4. 4

   Separate fan-out from work queues

   Inside each queue, several workers compete for messages. Scale them by queue depth and the age of the oldest message.

5. 5

   Handle failure

   Explain the visibility timeout for retries, a dead-letter queue after a few receives, and an alarm on the DLQ's visible message count.

6. 6

   Handle duplicates and order

   At-least-once delivery means idempotent consumers. Use a FIFO topic and FIFO queues with message groups only where order truly matters.

7. 7

   Name the costs

   Tracing gets harder, and data becomes eventually consistent, meaning other parts catch up a little later. Mention correlation ids and the metrics you would watch.