Design a Distributed Monitoring System: Metrics and Alerts

The one alert that matters gets buried under hundreds that do not. That is the problem monitoring system design solves: collect numbers from every machine, keep their history, and turn them into a few pages a person will act on.

Picture a common setup. A hosting business runs about 4,000 servers across two data centres in Hyderabad, and customers host their shop websites on them. The setup is made up, but the problem is real.

The old setup was a script on every server. Every five minutes it checked a few numbers and sent an email when any number crossed a line. Nothing kept history, and nothing decided which emails mattered.

One Tuesday at 3:10 a.m., a storage server filled its disk, and the websites on it started failing. At the same moment, a backup job pushed CPU use above 80 percent on 300 other servers. The on-call inbox, read by the engineer whose turn it is to respond, got 640 emails in twenty minutes.

• Most emails said CPU was high for one check, then normal again five minutes later.
• The disk email looked exactly like the CPU emails, so nobody noticed it.
• The script on the full server could not write its own files, so it went quiet.
• At 6:05 a.m., a customer phoned support to say their shop website was down.

The team did not lack data. They lacked a system that keeps history, notices silence and sends one clear page for one real problem. This guide designs that system, using Prometheus, a widely used open-source monitoring system, as the reference.

You need five words: metric, time series, label, scrape and alert rule. Each one is simple, and the rest of the guide builds on them.

Metric. A number that describes the system, measured again and again. Free disk space on a server is a metric. So is the count of web requests it has served since it started.

Time series. The list of values one metric has taken over time, each with a timestamp. Each pair of time and value is called a sample. Free disk on server store-12, one sample every 15 seconds, is one time series.

Label. A name and value pair attached to a metric, such as datacentre equal to hyd-1 or server equal to store-12. Labels let you filter and add up series, for example total errors per data centre.

Scrape. One visit by a collector to a server to read all its current metric values. If the collector scrapes every 15 seconds, each server is read four times a minute.

Alert rule. A saved question the system asks again and again, such as whether any disk is over 90 percent full. When the answer stays yes long enough, the rule fires an alert.

Monitoring system design is planning how to collect numbers from many machines, store them over time, show them on dashboards and alert people when something is wrong. The hard part is doing it at scale while staying cheap and quiet.

Think of a poultry farm supervisor in charge of forty sheds. Each shed has a thermometer. The supervisor walks a fixed round, writes each reading on a board beside the shed number, and phones the owner if a shed stays too hot.

Map it part by part. Each shed is a server, and a thermometer reading is a sample. The board's column of readings is a time series, and the shed number is a label. The round is the scrape, and the phone rule is the alert rule.

Where the picture breaks: the supervisor reads forty sheds, but a real collector reads millions of series every few seconds. And a board fills up, while a time series database compresses old readings and deletes them on a schedule.

What the system must do

• Collect metrics from 4,000 servers and the services on them every 15 seconds.
• Keep raw data for 15 days and summaries for a year, so the team can compare this Diwali sale week with last year's.
• Answer dashboard queries over the last hour in under a second.
• Evaluate alert rules every 30 seconds and page the right on-call engineer within minutes of a problem that lasts, while staying quiet about blips.
• Keep working when a data centre, a collector or the database has trouble, because that is exactly when people need it.

Notice the last point. A monitoring system must be more reliable than the things it watches. If it shares their fate, it goes blind exactly when you need it.

How much data is that?

Start with a quick estimate, because it drives every later choice. Suppose each server exposes about 500 series: CPU per core, memory, disk per mount, network per card and service counters. The Python below does the sums.

SERVERS = 4000
SERIES_PER_SERVER = 500
SCRAPE_SECONDS = 15
BYTES_PER_SAMPLE = 1.5    # after compression, roughly 1 to 2 bytes
RETENTION_DAYS = 15

series = SERVERS * SERIES_PER_SERVER
samples_per_sec = series / SCRAPE_SECONDS
bytes_per_day = samples_per_sec * 86_400 * BYTES_PER_SAMPLE

print(f"{series:,} series, {samples_per_sec:,.0f} samples per second")
print(f"{bytes_per_day / 1e9:.1f} GB per day")
print(f"{bytes_per_day * RETENTION_DAYS / 1e9:.0f} GB for {RETENTION_DAYS} days")

Run it with python3 and you should see 2,000,000 series and 133,333 samples per second, about 1.33 lakh (133,000). It then prints 17.3 GB per day and 259 GB for 15 days, in decimal gigabytes. We ran it on Python 3.13.16.

The line that matters most is BYTES_PER_SAMPLE. An uncompressed sample is a 64-bit timestamp plus a 64-bit value, so 16 bytes. The Prometheus storage docs say it averages 1 to 2 bytes per sample, and they size disks with this same formula.

So the bytes are modest. The real challenge is the rate: over 130,000 small writes every second that never stop. The estimate also leaves out the index and the write-ahead log, so leave headroom on disk.

A metrics pipeline has five main parts: agents on servers, collectors, an optional buffer, a time series database, and readers such as dashboards and alert rules. Follow the arrows in the diagram. Data flows one way, from servers to the on-call phone.

Agent. A small program on each server that reads local numbers, like disk space, and shows them as plain text on a web page. In Prometheus, the Node Exporter is the usual agent for machine numbers. Services such as the web hosting layer show their own counters too.

Collectors. Workers that visit every server on a timer and read its metrics. Each collector owns a slice of the servers, so adding servers means adding collectors.

Buffer. A queue between collectors and storage. If the database is slow for ten minutes, samples wait in the queue instead of being dropped. Some designs use Kafka, a durable message log, here. A single Prometheus server has no such queue, and small setups skip it.

Time series database. A database built for one pattern: huge numbers of small timestamped writes, and reads over time ranges. A later section shows how it stores them so cheaply.

Pull suits long-running servers, and push suits short jobs or machines the collector cannot reach. Prometheus pulls by default and offers a Pushgateway for the exceptions.

Back on the farm, there are two ways to get readings. In pull, the supervisor walks past every shed on a fixed round. In push, each shed worker phones the readings in. On the round, a shed with no reading stands out at once. With phone calls, a silent shed looks like a shed with nothing to report.

Service discovery means the collectors learn the current server list automatically, from an inventory database or a cloud platform. Without it, every new server needs a manual config change, and someone always forgets.

The fleet uses pull for its 4,000 servers. A nightly job that deletes closed customer accounts across the whole service pushes its result to a Pushgateway. The collectors then scrape the gateway like any other target.

Per-server jobs are different. The Prometheus docs say machine-level batch jobs should write their numbers to a file that the Node Exporter's textfile collector reads, not to the Pushgateway. So each server's backup script writes its last success time there.

Why the care? The docs list the Pushgateway's traps. It never forgets a series pushed to it, and one gateway serving many jobs becomes a single point of failure. Pushed jobs also lose the automatic up metric that tells you a target is dead.

Alerting works in two stages: a rule evaluator decides when a problem has lasted long enough to matter, and an alert manager decides who hears about it and how. Keeping the stages apart is what stops a flood of 640 emails.

Stage 1: rules that wait before they fire

Every 30 seconds, the rule evaluator runs each rule as a query on recent data. When a condition first becomes true, the alert is pending, which means seen but not yet sent. It becomes firing only if the condition stays true for a set time, set by the rule's for clause.

If the condition turns false even once while the alert is pending, Prometheus drops the pending alert. The clock then starts again from zero next time. The diagram shows this for the disk rule.

Here is that waiting logic in plain Python. Each call passes the latest rule results: a dictionary from each series' labels, kept as a tuple, to its disk use from 0 to 1. The loop at the bottom plays ten minutes of evaluations.

FOR_SECONDS = 300          # condition must hold for 5 minutes
pending_since = {}         # series labels: when it first went bad

def evaluate(results, now):
    firing = []
    for labels in list(pending_since):
        if labels not in results:          # series vanished
            del pending_since[labels]
    for labels, used in results.items():
        if used <= 0.90:                   # healthy again
            pending_since.pop(labels, None)
            continue
        start = pending_since.setdefault(labels, now)
        if now - start >= FOR_SECONDS:
            firing.append((labels, used))
    return firing

spike = (("server", "web-07"),)    # labels kept as a tuple of pairs
full = (("server", "store-12"),)

for now in range(0, 601, 30):      # one evaluation every 30 seconds
    results = {full: 0.95}
    if now <= 120:                 # web-07 is over 90 percent for 2 minutes
        results[spike] = 0.93
    else:
        results[spike] = 0.60
    firing = evaluate(results, now)
    if firing:
        print(now, "seconds:", [dict(labels)["server"] for labels, _ in firing])
        break

Run it and you should see one line: 300 seconds: ['store-12']. We ran it on Python 3.13.16. Server web-07 never pages, because its healthy reading at 150 seconds cleared its timer.

What just happened? The setdefault line records the first time a series went bad and keeps that time on later calls. A healthy reading pops the entry, so the five-minute clock restarts. The first loop also forgets a series that stops reporting, as Prometheus does.

That last point hides a trap. A rule about disk use cannot fire for a server that has gone silent. So you need a separate rule that watches the up metric, which is 0 when a scrape fails.

Stage 2: the alert manager

The alert manager receives firing alerts and decides what reaches a human. In Prometheus this is a separate program called Alertmanager. It does five jobs, and each one fixes part of that bad night.

• Deduplication: the same alert sent by two rule evaluators becomes one notification, as long as both send identical labels.
• Grouping: alerts that share chosen labels, such as all disk alerts in hyd-1, go out as one message listing every server.
• Inhibition: if a whole data centre is unreachable, alerts for single servers inside it are muted.
• Silences: during planned maintenance, an engineer mutes matching alerts for a set time.
• Routing: labels decide the receiver, so storage alerts reach the storage on-call and minor ones become tickets.

Inhibition is the farm supervisor again. If the power fails for a whole row of sheds, he does not report forty separate fan failures. He reports one power cut.

It appends new samples to an in-memory head block protected by a write-ahead log, then writes compressed two-hour blocks to disk. It almost never changes old data in place. This section follows Prometheus 3.15.0, released on 24 September 2026 and the latest release as of 19 October 2026.

Metric data is unusual. New samples always arrive at the end, old samples never change, and queries ask for ranges like the last six hours. A storage engine built for exactly this can skip most of the work a general database does.

1. 1

   Write to the log first

   Each sample is appended to the write-ahead log, a file that records changes before they are applied. The head is not fully saved to disk, so after a crash Prometheus replays this log at restart.

2. 2

   Keep the head block

   Recent samples live in the head block. Dashboards mostly ask about the last hour or so, which the head answers without opening any block files.

3. 3

   Cut a two-hour block

   Samples are grouped into two-hour blocks. Each block is a directory of compressed chunks, a metadata file and an index from labels to series.

4. 4

   Merge small blocks

   A background job called compaction merges blocks into bigger ones, up to 10 percent of the retention time or 31 days, whichever is smaller.

5. 5

   Drop data past retention

   Whole blocks past the retention period, 15 days by default, are deleted in one go. The docs warn this can take up to two hours.

Compaction is why a week-long query stays quick. A week holds 84 two-hour blocks. With 15-day retention the cap is 36 hours, and the v3.15.0 source merges blocks in steps of 2, 6 and 18 hours. So the largest block spans 18 hours, and a week-long query opens about a dozen blocks instead of 84.

Why compression works so well

Metric data is boring in a useful way. Prometheus's chunk format follows a design from the Gorilla paper, published by Facebook engineers in 2015. Two tricks do most of the work.

• Timestamps: scrapes arrive every 15 seconds, so the gap barely changes. The database stores the change in the gap, and an unchanged gap costs one bit.
• Values: each value is XORed with the previous one, a bit-level comparison. An identical value costs one bit, and a slow-changing value stores only the bits that differ.

On Facebook's own data, the paper reports an average of 1.37 bytes per sample, down from 16. About 96 percent of timestamps and 51 percent of values compressed to a single bit. Your own ratio depends on how noisy your values are.

Keeping a year of history cheaply

Nobody needs 15-second detail from last March. Downsampling adds summaries per window for old data, so a query over a year reads far fewer samples. Thanos, an open-source layer for long-term Prometheus storage, builds 5-minute summaries after 40 hours and 1-hour summaries after 10 days.

Each summary keeps the count, sum, minimum and maximum of its window, so you can still get the average. Keep the maximum in mind. A disk that hit 100 percent for ten minutes looks fine in an hourly average.

Here is the catch. The Thanos docs say downsampling does not save space by itself, because it adds summary blocks next to the raw ones. As a rule of thumb, they suggest equal retention at every level. The team trades that away to save storage: raw samples for 15 days, 5-minute summaries for 90 days, 1-hour summaries for a year. The cost is that it cannot zoom into raw detail older than 15 days.

Spreading the load across machines

The Prometheus docs say local storage is not clustered or replicated. So a fleet this size sends samples on to a remote store built from many machines. Grafana Mimir, one open-source option, hashes each series to pick which machines hold it.

In Mimir's classic architecture, each series gets three copies by default; its newer ingest storage architecture leaves copies to Kafka. The cost is that a query for total errors must ask many machines and add up the answers.

High cardinality means a metric has a huge number of label combinations. It breaks monitoring because every combination is a separate time series to index, keep in memory and query. Cardinality just means the count of distinct values.

Here is how it happens. A developer adds a customer_id label to the web request counter to see traffic per customer. The counter already has labels for status code and HTTP method.

Before the change, one web server had about 5 status codes times 4 methods, so 20 series for that counter. Behind a load balancer, which spreads requests across servers, any server can serve any of 12,000 customers. The same counter can now reach 240,000 series per server, and the database's memory climbs until it falls over.

Guards, and the trap inside one

The team adds two guards. First, each scrape gets a sample_limit of 5,000, ten times the normal 500 series. Second, a weekly report lists the metrics whose series count grew fastest.

Now the trap. The configuration docs say a scrape over sample_limit is treated as failed as a whole. In the v3.15.0 source, that sets the server's up metric to 0 and marks all its series stale.

So one bad label makes a healthy server look dead, and its disk and CPU numbers vanish too. Alert on prometheus_target_scrapes_exceeded_sample_limit_total as well, so the page says what really happened.

With this guide's settings, a lasting disk problem pages about six minutes after it starts. Each stage adds its own delay, and most of them are deliberate.

So the answer is about 3:16:15 a.m. That assumes the disk alert starts a new group. If its group is already firing, it joins the next group_interval send instead, up to 5 minutes later. Each later notification for the same group waits group_interval, 5 minutes by default. A repeat of the same message waits repeat_interval, 4 hours by default.

You can shorten any stage, but each cut buys noise. A shorter for clause pages on blips, and a shorter group_wait splits one outage into several pages.

Page only for problems that are urgent, real and need a human, and alert on what users feel instead of every internal cause. Alert fatigue is when people get so many alerts that they stop reading them.

The Google SRE book, written by Google's site reliability engineers, names four golden signals. It says that if you can measure only four metrics of a user-facing system, measure these.

• Latency: how long requests take. Track slow percentiles, not just the average, because an average can hide a slow tail.
• Traffic: how much demand there is, such as requests per second.
• Errors: the rate of requests that fail, including ones that return a wrong answer.
• Saturation: how full the service is, such as disk, memory or connection slots.

High CPU on its own is a cause, not a symptom. On that bad night, 300 servers were busy with backups, but no customer was hurt. Those alerts should have been dashboard lines, not emails.

Some causes deserve a page because the harm is close and certain. A disk that will be full within four hours is one. As of 19 October 2026, the node exporter's published alert rules handle it. They mark an alert critical when free space is under 20 percent and predict_linear says it hits zero within 4 hours.

predict_linear fits a straight line through the last six hours of free space and extends it forward. That rule also has a for clause of one hour, so a brief burst of writes does not page.

Alerting on the error budget

An SLO, or service level objective, is a target such as 99.9 percent of requests succeeding over 30 days. The 0.1 percent left over is the error budget. Burn rate is how fast you spend it.

A burn rate of 1 uses exactly the whole budget in 30 days. A burn rate of 14.4 held for one hour uses 2 percent of it. The Google SRE workbook recommends three alerts for a 99.9 percent target.

• Page if the burn rate is above 14.4 over the last hour and also over the last 5 minutes. That is 2 percent of the budget.
• Page if it is above 6 over the last 6 hours and also over the last 30 minutes. That is 5 percent.
• Open a ticket if it is above 1 over the last 3 days and also over the last 6 hours. That is 10 percent.

Why two windows each? The long window proves the problem is big enough to matter. The short window proves it is still happening, so the alert stops firing soon after the errors stop.

When the monitoring system fails, nothing pages, so you must design it to fail loudly. The usual answer is two copies of every alerting part, plus a simple outside checker that expects a regular heartbeat.

Remember the full server whose script went silent. Silence is the most dangerous signal in monitoring, because it looks exactly like everything being fine. The diagram shows how the team guards against it.

Both rule evaluators send every alert to both alert managers. The Alertmanager docs warn against putting a load balancer between Prometheus and its Alertmanagers; point Prometheus at all of them. The managers form a cluster, so one alert usually pages once. Prometheus adds its external labels to every alert it sends. If the two evaluators differ in a replica label, drop it with alert_relabel_configs, or Alertmanager sees two different alerts.

The heartbeat is an alert rule that is always true. The prometheus-operator runbooks call theirs Watchdog and say it should always be firing. The team sets both group_interval and repeat_interval to 1 minute on the Watchdog route. Alertmanager checks repeat_interval only at each group_interval, so a short repeat_interval alone still sends only every 5 minutes, the default group_interval. With both at 1 minute, the outside checker hears it every minute or two.

If the outside checker stops hearing it, something in the chain is broken, and it pages directly. This pattern is often called a dead man's switch.

Three months later, a disk filled again on a storage server. The fill-time rule paged the on-call engineer about three hours ahead, in one message, and the team fixed it before any customer noticed.

Explain it as a pipeline with an alerting path on top: agree the scale first, then walk from collection to storage to alerts, naming one trade-off at each step. Interviewers want to see that you think about cost and noise, not just boxes.

1. 1

   Pin down the scale

   Ask how many servers, series per server, scrape interval and retention. Turn them into samples per second and storage per day, out loud.

2. 2

   Draw the pipeline

   Agents, collectors, an optional buffer, a time series database, a query layer, then the rule evaluator and alert manager.

3. 3

   Choose pull or push

   Pull for long-running servers, a push gateway for service-level batch jobs. Mention service discovery so new servers appear automatically.

4. 4

   Explain the storage

   Write-ahead log, in-memory head, compressed two-hour blocks, compaction and downsampling. Shard by a hash of series labels and keep copies.

5. 5

   Name the cardinality risk

   Each label combination is a new series. Cap samples per scrape, say what happens when the cap trips, and keep IDs out of labels.

6. 6

   Make alerts trustworthy

   Rules with a for clause, grouping, inhibition and routing. Alert on symptoms and error budget burn rate, and state the end-to-end delay.

7. 7

   Close with self-monitoring

   Two copies of each alerting part, no load balancer in front of the alert managers, and an outside heartbeat checker.