Notification System Design: Push, SMS and Email, Done Right

Messages arrive late, arrive twice, or arrive after someone switched them off. When every service calls the SMS and email companies itself, each makes its own choices and nobody sees the whole picture. Notification system design fixes this with one shared service that every other part of the app goes through.

Picture a common setup. A home-services booking app in Kochi connects customers with electricians, carpenters, cleaners and AC repair technicians. It tells people what is happening: booking confirmed, technician on the way, job done, bill paid. The setup is made up, but the problems are real.

On a normal day the app sends about 3 lakh (300,000) notifications to its customers and 4,000 technicians. In the old setup, the booking, payment and marketing services each called the SMS and email providers on their own.

Then came the Onam sale, timed for Kerala's harvest festival. The marketing team scheduled 20 lakh offer messages to past customers for 7 pm, 2 lakh of them by SMS. The next morning, support had four complaints.

• A customer waited about half an hour for the job-start OTP (one-time password) her electrician needed. It was stuck behind the sale messages in the same SMS queue.
• Customers who had switched off offers in the app still got offer SMS. The marketing code checked only the push setting.
• One customer got the same technician-arriving SMS four times. A timeout made the booking service try again, and nothing noticed the repeats.
• About 18 percent of push sends went to device tokens that no longer worked, mostly from phones where the app had been removed. The old code ignored the error that came back.

Each complaint has a different cause, and each maps to one part of the design below. The rest of this guide rebuilds the app's notifications to fix all four.

Notification system design is planning one shared service that turns events from other systems into messages people receive. The hard parts are not the sending. They are choosing the channel, choosing the time, and retrying without sending twice.

Six words carry the rest of this guide.

• Channel: a way to reach someone. This app uses three: push notifications, SMS and email.
• Push notification: a short alert the phone's operating system shows, even when the app is closed.
• Event: a short fact another service reports, such as technician_assigned for booking 88213. The booking service does not choose wording or channels. It only says what happened.
• Provider: the outside service that delivers the message. For SMS it is an SMS gateway company. For push it is the push service built into iPhones, or Firebase Cloud Messaging (FCM), Google's push service, on Android.
• Device token: the address of one app on one phone, issued by the push service. It stops working when, for example, the person uninstalls the app.
• Template: fixed wording with blanks, like Your {service} expert arrives by {time}. The team can edit templates in English and Malayalam without a code change.

Think of a family announcing a baby's naamkaran, the naming ceremony. Grandmother gets a phone call, because she never reads messages. Cousins abroad get a text with photos. A great-uncle who distrusts phones gets a printed card.

Map it part by part. The news is the event. Each relative's preferred way to hear is a preference. The call, the text and the card are channels. Whoever makes the calls and posts the cards is a worker.

Where the picture breaks: a family shares news once, with a few dozen people who answer the phone. A notification service handles lakhs of events a day, through providers that time out, fail and sometimes repeat themselves.

A notification system must send push, SMS and email from one entry point, respect each person's choices, and keep urgent messages fast. It must never lose a message silently, and it must avoid duplicates.

What it must do

• Accept events from any service in the app through one API (the entry point other programs call).
• Send by push, SMS and email, using templates in English and Malayalam.
• Respect preferences per category and per channel, plus quiet hours at night.
• Schedule messages, like a reminder one hour before a booking.
• Record every attempt and its result, so support can see what each customer received.

How well it must do it

• Speed: an OTP reaches the provider within 2 seconds of the event, even during a campaign. That is this app's own target, not an industry rule.
• Reliability: no accepted event is lost. Anything that cannot be sent ends up somewhere a person can see it.
• Scale: about 3 lakh messages on a normal day, and 20 lakh on a campaign evening.
• Cost: SMS is the most expensive channel, so it is used only when push will not do.

Rough numbers

Three lakh a day is about 3.5 messages per second on average (300,000 divided by 86,400 seconds). The dinner-time peak might be ten times that, about 35 per second. The Onam campaign is the real test: 20 lakh in two hours is about 280 per second.

Storage is small. At about 1 KB per delivery record, 3 lakh records a day is about 300 MB, or roughly 9 GB a month. One database handles that easily.

A notification system works as a pipeline. An API accepts the event, a planner checks preferences and fills templates, queues hold the jobs, and channel workers hand them to providers. Each stage can slow down without breaking the others.

Follow the technician_assigned event in the diagram below, from the booking service down to the three workers. Notice that only the workers talk to outside providers, and all of them write to one delivery log.

1. 1

   Accept and save

   The notification API checks the event, gives it a notification id and saves it. Then it replies within milliseconds, so the booking service never waits for an SMS.

2. 2

   Plan

   The planner reads the customer's preferences. For technician_assigned, this customer wants push, with SMS only if push fails.

3. 3

   Fill the template

   It fills the template in the customer's language, for example: Your electrician arrives by 4:30 pm.

4. 4

   Queue

   It puts one job per channel on the right priority lane. A job carries the notification id, the channel, the address and the text.

5. 5

   Send and record

   A worker for that channel takes the job, calls the provider, and writes the result to the delivery log.

Why put a queue in the middle? A queue is a list of jobs waiting their turn, kept by a separate system such as Kafka, RabbitMQ or Amazon SQS. If the SMS gateway slows down, jobs wait safely in the queue instead of piling up inside the API.

Each channel has its own provider, address, cost and way of failing, so each gets its own worker. Push needs a live device token. SMS reaches almost any phone but costs the most. Email suits long receipts.

Push: keep the token list clean

Push has the most moving parts, because the address keeps changing. The app gets a device token from the phone's push service and sends it to the app's servers.

FCM's documentation recommends saving a timestamp with each token and updating it on every upload. By default, FCM calls a token stale when its app has not connected for a month.

On Android, FCM treats a token inactive for 270 days as expired. If FCM answers UNREGISTERED or INVALID_ARGUMENT to a message whose content you know is valid, its documentation says you can delete that token. That clean-up fixes the 18 percent waste.

As of October 2026, FCM's token guide also says FCM is moving from registration tokens to Firebase Installation IDs (FIDs), with both supported during the change. The idea stays the same: one address per app per phone. Check the current guide before you build.

Push: replace stale alerts and set a lifetime

Push can replace an old message with a new one. The web push standard, RFC 8030, has a Topic header for this. FCM has a collapse key, and allows at most four different collapse keys per device at a time.

A phone that was offline then gets only the latest update: arriving in 5 minutes, not also arriving in 20 minutes.

Every push message also has a lifetime, called TTL (time to live): how long the push service keeps it for an offline phone. RFC 8030 counts it in seconds. On FCM, a message sent without a TTL is kept for the maximum, four weeks (2,419,200 seconds).

So the app sets a short TTL on arrival alerts, such as 15 minutes (900 seconds). A technician-arriving alert that shows up next week only confuses people.

SMS: the expensive, reliable fallback

SMS goes through an SMS gateway, a company connected to the mobile operators. It reaches phones without the app, which makes it the right channel for OTPs. In this app it is also the most expensive channel by far.

In India, TRAI, the telecom regulator, requires a business sending commercial SMS to register itself as a principal entity. It must also register its sender names (called headers) and its message templates. TRAI's advice page says failing any step stops you from sending.

Links need one more step. TRAI's press release of 30 August 2024 covers links to websites (URLs), app install files (APKs) and other apps (OTT links). From 1 October 2024, traffic with such links is not permitted unless they are whitelisted. So register every link your templates use, such as a booking-tracking page, before a campaign goes out.

The gateway's first reply only means it accepted the message. The real result comes later as a delivery report, sent to a webhook: a web address on your server that the gateway calls.

Email: handle bounces and unsubscribes

Email goes through an email sending service, and two things matter most: bounces and unsubscribes. A hard bounce means the address does not exist, so stop using it. A soft bounce, like a full mailbox, can be tried again later.

For offer emails, RFC 8058 defines one-click unsubscribe. The email carries a List-Unsubscribe header with an HTTPS address, and a List-Unsubscribe-Post header set to List-Unsubscribe=One-Click. When the person taps unsubscribe, their mail service sends an HTTPS POST, a secure web request, to that address.

That request alone must do the job. The standard says the process has to work without manual intervention, and the sender must not answer with a redirect. Both headers must be covered by a valid DKIM signature, which proves an email really came from the sender's domain.

For big senders this is no longer optional. Gmail's sender guidelines say that from 1 February 2024, senders of more than 5,000 messages a day to personal Gmail accounts must support one-click unsubscribe on marketing messages.

To check your own emails, open the raw source of an offer email in your mail app. You should see List-Unsubscribe-Post: List-Unsubscribe=One-Click, and the authentication results should show dkim=pass. Then find the DKIM-Signature header: its h= list names the signed headers and must include List-Unsubscribe and List-Unsubscribe-Post. The service records each one-click unsubscribe in the same preferences store the planner reads.

Preferences are stored rules about which kinds of messages a person wants, on which channels, and at what times. The planner reads them before creating any job, so a muted channel never reaches a worker.

The team puts every template into one of three categories. Critical covers OTPs and security alerts. Transactional covers booking and payment updates. Promotional covers offers and campaigns.

Each category follows different rules. Critical messages skip opt-outs, because the person needs them right now. Transactional messages follow the person's channel choices. Promotional messages also follow quiet hours, and need the person's consent first.

Back to the naamkaran. Grandmother has said: call me, but never after 9 at night. So when the news is ready at 10 pm, you wait and call her at breakfast. Quiet hours are that same promise, written into code.

Here is the planner's check as a runnable sketch. Prefs holds the channels a person chose for each category, and the channels that have a working address. A category the person never agreed to has no channels, so nothing is sent.

from dataclasses import dataclass, field
from datetime import datetime, time, timedelta

QUIET_START, QUIET_END = time(21, 0), time(8, 0)    # 9 pm to 8 am

@dataclass
class Prefs:
    chosen: dict                                    # category -> channels picked
    working: set = field(default_factory=set)       # channels with a live address

def plan_delivery(prefs, category, now):
    """Return (channels, send_at). now is the person's local time."""
    if category == "critical":                      # OTPs, security alerts
        return [c for c in ("sms", "push") if c in prefs.working], now
    channels = [c for c in prefs.chosen.get(category, []) if c in prefs.working]
    t = now.time()
    if category == "promotional" and (t >= QUIET_START or t < QUIET_END):
        day = now.date() + timedelta(days=1 if t >= QUIET_START else 0)
        return channels, datetime.combine(day, QUIET_END)
    return channels, now

prefs = Prefs(chosen={"transactional": ["push"], "promotional": ["push"]},
              working={"push", "sms"})
at_10pm = datetime(2026, 10, 19, 22, 0)
for category in ("critical", "transactional", "promotional"):
    channels, send_at = plan_delivery(prefs, category, at_10pm)
    print(category.ljust(14), str(channels).ljust(16), f"{send_at:%d %b %H:%M}")

Save it as plan_delivery.py and run python3 plan_delivery.py. You should see this, with times in the person's local time:

critical       ['sms', 'push']  19 Oct 22:00
transactional  ['push']         19 Oct 22:00
promotional    ['push']         20 Oct 08:00

What just happened? At 10 pm the OTP goes out at once on both working channels, ignoring choices and quiet hours. The booking update goes out at once on push only. The offer is not dropped: it is held until 8 am the next morning, and a scheduler releases it then.

This sketch uses one fixed quiet window. A real system stores quiet hours per person and works them out in that person's time zone.

This one check fixes the second complaint. No service can forget the SMS setting, because no other service sends SMS at all.

Retry only temporary errors, wait longer before each new attempt with some randomness, and stop after a fixed number of attempts. Claim the job in a delivery log first, so a retry never sends a second copy.

Temporary errors may pass on their own: a timeout, a provider saying too many requests, or a server error like 503. Permanent errors will not pass: an invalid number, a dead token, a hard bounce. Retrying a permanent error only wastes money.

Providers often say exactly how to retry. FCM's error guide says to honour the Retry-After header on a 503 and to use exponential backoff, with jitter when sending many messages. For some 429 errors (too many requests), such as an overall send rate that is too high, it asks for backoff starting at 1 minute or more.

Backoff and jitter

Backoff means waiting longer after each failure: 2 seconds, then 4, then 8. Jitter means picking a random wait up to that limit. Without it, thousands of jobs that failed together all retry in the same second and fail together again.

Marc Brooker's 2015 article on the AWS Architecture Blog simulated clients competing for one resource. With 100 competing clients, adding jitter more than halved the number of calls compared with backoff alone. He called backoff without jitter "the clear loser" of the approaches he compared.

Duplicates and the delivery log

Amazon SQS standard queues, for example, promise at-least-once delivery, and their documentation warns that more than one copy of a message might be delivered. A worker can also crash after the provider accepted a message but before saving the result, so the job runs again.

So every job carries its notification id, and the worker checks the delivery log first. This property is called idempotency: running the same job twice has the same effect as running it once.

A plain check is not enough. Two copies of a job can run on two workers at the same moment. Both read the log, both see nothing, and both send.

So the check and the claim must be one atomic write, one that happens fully or not at all. The handler below uses an upsert, an insert that becomes an update when the row already exists. The write succeeds only for a new id, or for a job waiting to retry.

import random, sqlite3

MAX_ATTEMPTS, BASE, CAP = 5, 2, 300        # BASE and CAP in seconds

class PermanentError(Exception): pass      # bad number, dead token
class TemporaryError(Exception):           # timeout, 429, 503
    def __init__(self, msg, min_wait=0):
        super().__init__(msg)
        self.min_wait = min_wait           # from a Retry-After header

log = sqlite3.connect(":memory:")          # the delivery log
log.execute("CREATE TABLE delivery_log (id TEXT PRIMARY KEY, status TEXT)")

def claim(job_id):
    """Take the job in one atomic write. Only one copy can win."""
    cur = log.execute("INSERT INTO delivery_log VALUES (?, 'sending') "
                      "ON CONFLICT(id) DO UPDATE SET status = 'sending' "
                      "WHERE status = 'retry'", (job_id,))
    log.commit()
    return cur.rowcount == 1

def mark(job_id, status):
    log.execute("UPDATE delivery_log SET status = ? WHERE id = ?", (status, job_id))
    log.commit()

def handle(job):
    if not claim(job["id"]):               # already sent, or being sent
        return "skipped repeat"
    try:
        send_to_provider(job)
        mark(job["id"], "sent")
        return "sent"
    except PermanentError:
        mark(job["id"], "failed")
        return "failed for good"
    except TemporaryError as err:
        if job["attempt"] + 1 >= MAX_ATTEMPTS:
            mark(job["id"], "dead")
            dead_letters.append(job)
            return "dead-lettered"
        mark(job["id"], "retry")
        limit = min(CAP, BASE * 2 ** job["attempt"])
        wait = max(err.min_wait, random.uniform(0, limit))
        retry_queue.append(({**job, "attempt": job["attempt"] + 1}, wait))
        return f"retry, backoff limit {limit} s"

# A fake gateway that fails twice, then accepts
calls, retry_queue, dead_letters = [], [], []
def send_to_provider(job):
    calls.append(job["attempt"])
    if len(calls) <= 2:
        raise TemporaryError("503 Service Unavailable")

job = {"id": "booking-88213:technician_assigned:sms", "attempt": 0}
print(handle(job))
print(handle(retry_queue.pop()[0]))
print(handle(retry_queue.pop()[0]))
print(handle(job))                         # the queue hands over a copy again
print("provider calls:", len(calls))

Save it as handler.py and run python3 handler.py. It needs only Python's built-in SQLite. You should see:

retry, backoff limit 2 s
retry, backoff limit 4 s
sent
skipped repeat
provider calls: 3

What just happened? The gateway was called three times: two failures and one success. The repeated copy was skipped, because its row already said sent. That is the four-copies complaint fixed.

The wait line is backoff with full jitter: a random wait between zero and a limit that doubles each attempt. With five attempts the limits are 2, 4, 8 and 16 seconds, so backoff adds at most 30 seconds unless the provider asks for longer. The 300-second cap only matters if you allow ten or more attempts.

If the provider asks for a longer wait, min_wait wins. For an FCM 429 you would set it to at least 60 seconds. A real worker would put the job back on the queue with that delay, instead of the list used here.

One small gap remains. If a worker crashes after the provider accepts but before the row is marked sent, the row stays at sending. A sweeper job must later decide whether to resend, so a rare duplicate is still possible.

If your provider accepts your own reference id and ignores repeats of it, pass the notification id and the provider closes that gap. Otherwise, accept the rare duplicate and say so in interviews.

Dead-letter queue and fallback

A dead-letter queue is a separate queue for jobs that ran out of attempts. Nothing is lost: an engineer can read them, fix the cause, and send them back.

Amazon SQS, for example, moves a message to its dead-letter queue after consumers have received it maxReceiveCount times. Its documentation advises giving the dead-letter queue a longer retention period (how long a message is kept) than the original queue.

The reason is a trap. In a standard queue, a message's age counts from when it first entered the source queue, and the move does not reset that clock. In a FIFO queue the clock does reset.

Take the documentation's own example. A message spends 1 day in the source queue, and the dead-letter queue keeps messages for 4 days. In a standard queue it is deleted from the dead-letter queue after 3 more days, not 4.

If you use SQS for the retry delay itself, one message can be delayed by at most 15 minutes. FIFO queues do not support delays on single messages at all. Longer waits need a scheduler.

For arrival alerts, the team adds one more step, shown in the diagram below. When a push job lands in the dead-letter queue, a handler sends the same alert by SMS instead. Start at the top box and follow the No arrows.

Give each priority its own queue and its own workers, and limit how fast the promotional lane sends. A campaign then fills only its own lane, while OTPs pass through an almost empty one.

That is what went wrong on Onam. The 2 lakh offer SMS all joined one shared SMS queue at 7 pm. At about 100 SMS a second (made up), 1.7 lakh were still waiting five minutes later.

An OTP added at 7:05 pm therefore waited 170,000 divided by 100, which is 1,700 seconds or about 28 minutes. That is the half-hour complaint.

The diagram below shows the fix. Look at the promotional lane: it is the only one that passes through a rate limiter (a gate that allows only so many sends per second) before reaching the workers.

Lanes alone may not be enough. If your gateway caps sends per second for your account, offers can use the whole cap, and OTPs still wait at the gateway. So check your gateway contract, and reserve part of the cap for the critical lane.

The marketing team changed the campaign too. The next sale goes out over three hours, one Kochi locality at a time, instead of all at 7 pm.

To check the fix, measure OTP time from event to provider during the next campaign. With its own lane and reserved capacity, it should stay under the 2-second target while the promotional lane drains.

Store a status for every notification on every channel, update it from provider replies and delivery reports, and watch failure rates per channel. Sent is not the same as delivered.

• Queued: the job is waiting in its lane.
• Sent: the provider accepted it.
• Delivered: the provider confirmed it reached the phone or the mailbox.
• Failed: it could not be sent, with the reason saved.
• Opened: the person tapped the push or opened the email, where you track this.

Delivery reports arrive late, and sometimes out of order. A late Sent update must never overwrite Delivered. So in this delivery log, a status only moves forward, from Queued to Sent to Delivered, never back.

The on-call dashboard shows three numbers: OTP time from event to provider, failure rate per channel, and dead-letter queue size. A jump in any of them alerts the on-call engineer.

Walk from requirements to the pipeline, then spend most of your time on priorities, retries and duplicates, where interviewers look for depth.

1. 1

   Clarify message types

   Ask which messages are urgent, which channels exist, and the daily volume. Split critical, transactional and promotional messages early.

2. 2

   Estimate

   Turn daily volume into messages per second, average and peak. Note the campaign peak and the provider's rate limit.

3. 3

   Draw the pipeline

   API, planner with preferences and templates, priority queues, one worker pool per channel, and a delivery log.

4. 4

   Handle failure

   Separate temporary and permanent errors. Add backoff with jitter, an attempt limit, a dead-letter queue and a fallback channel for important alerts.

5. 5

   Handle duplicates

   Say that queues deliver at least once, so workers claim the notification id in the delivery log with one atomic write. Admit the small crash gap that remains.

6. 6

   Close with operations

   Mention delivery tracking, dashboards, dead token clean-up, short push lifetimes for time-bound alerts, and cost per channel.