Design a Distributed Logging System

The line you need sits on a machine you cannot find, or on one that no longer exists. With two servers, logging in to each one works. With dozens of servers whose containers come and go, it stops working. Distributed logging is how teams fix that.

Picture a common setup. A payments app for small shops runs six services as 300 containers (small packaged copies of a program) on 40 servers. The setup is made up, but the problem is real.

Kubernetes, the system that runs those containers, moves them between servers all day. One Friday a shop owner calls support. A customer paid Rs 2,340 and the money left the bank, but the app still shows the payment as pending.

The on-call engineer logs in to servers one by one and searches with grep, a command that finds text in files. Two hours later she finds the bank adapter's line. The ledger's line is gone, because its pod was evicted at 6 pm.

That is not bad luck. A pod is the small group of containers Kubernetes schedules together. The Kubernetes docs say that when a pod is evicted from a node (one server in the cluster), its containers go too, along with their logs.

So the team writes a wish list. Every line from every service in one place. Searchable in seconds. Kept for a year for audits. Safe even when a container dies. That wish list is a distributed logging system.

Distributed logging is a system that collects log lines from many machines, moves them over the network to central storage, and lets you search them in one place. People also call it centralized logging or log aggregation.

Think of a police thana with ten beat constables. Each constable writes what he sees in a pocket diary, with the time beside every entry. At the end of the shift, a jeep brings all the diaries back to the thana, where a writer files them by beat and date.

Map it part by part. The servers are the constables, and log lines are diary entries. The agent that collects them is the jeep, and the central store is the thana's shelves. When the officer in charge asks about the market road at 9 pm, nobody chases ten constables.

Where the picture breaks: diaries come back once a shift, but log lines leave every second or so. And a diary keeps its old pages, while a server rotates its log files away.

The team turns its wish list into requirements:

• Collect from every container, including ones that live for only a few minutes.
• Lose almost nothing, and never lose errors or payment events without knowing it.
• Search across all services by time, service, level and id, with results in seconds.
• Keep the last few weeks fast to search and the rest of the year cheap to store.
• Keep card numbers, OTPs and passwords out of every copy.
• Never slow down a payment because logging is slow.

A good log line is structured: a small JSON object with fixed fields such as time, level, service, ids and message. Machines can then filter on fields instead of guessing at free text.

Compare two versions of the same ledger event. The free-text version is a sentence saying payment pay_8f3a21 for merchant M-4417 failed after 3 tries. The structured version puts each of those values in its own named field.

With free text, a search for merchant M-4417 must match words inside sentences, and every team phrases them differently. With fields, you ask for merchant_id equal to M-4417 and get exactly those lines, from every service.

The level says how serious a line is. DEBUG is detail for developers, INFO is a normal event, WARN is something odd, and ERROR is a failure. The OpenTelemetry log data model, an open standard, maps levels to severity numbers 1 to 24, from TRACE up to FATAL.

Two ids on every line

The most useful fields are ids. A request id is created when a request first arrives and is passed to every service that request touches. A business id, such as the payment id, ties together different requests about the same payment.

You need both because one payment is several requests. The bank's callback is a new request that the bank starts, so it arrives without your request id. Only the payment id links it to the shop's first request.

If you use distributed tracing, its trace id plays the request id's role. The W3C Trace Context standard carries it in a traceparent header, where the trace id is 32 hex characters (0 to 9 and a to f). OpenTelemetry log records have a TraceId field for it.

A structured logger in Python

Here is how the settlement service writes structured lines with Python's standard logging module. The request id lives in a context variable, a value Python keeps separate for each request, even when many requests share one thread in async code.

import json, logging, sys
from contextvars import ContextVar
from datetime import datetime, timezone

request_id = ContextVar("request_id", default="-")

class JsonFormatter(logging.Formatter):
    def format(self, record):
        ts = datetime.fromtimestamp(record.created, timezone.utc)
        line = {"ts": ts.isoformat(timespec="milliseconds").replace("+00:00", "Z"),
                "level": record.levelname, "service": "settlement",
                "request_id": request_id.get(), "msg": record.getMessage()}
        for key, value in getattr(record, "fields", {}).items():
            line.setdefault(key, value)  # extra fields never overwrite the core ones
        if record.exc_info:  # log.exception adds the traceback
            line["exc"] = self.formatException(record.exc_info)
        return json.dumps(line)

handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(JsonFormatter())
log = logging.getLogger("settlement")
log.addHandler(handler)
log.setLevel(logging.INFO)

request_id.set("req_7c41e9")
log.info("upi callback received", extra={"fields": {
    "payment_id": "pay_8f3a21", "merchant_id": "M-4417", "amount_paise": 234000}})

Three parts matter. The formatter turns every record into one JSON line with the same core fields, and extra fields cannot overwrite them. The call to request_id.get() adds the id, so nobody passes it by hand. The handler writes to standard output, not to a file.

The if record.exc_info line matters for errors. When you call log.exception, it adds an exc field holding the traceback, Python's list of the calls that led to the error.

{"ts": "2026-10-19T09:15:42.318Z", "level": "INFO", "service": "settlement", "request_id": "req_7c41e9", "msg": "upi callback received", "payment_id": "pay_8f3a21", "merchant_id": "M-4417", "amount_paise": 234000}

What just happened? One call produced one line that any tool can filter by payment_id or merchant_id. In our test, 100 async tasks each set their own request id, and none of them read another task's value, because each task gets its own copy of the context.

Writing to standard output follows the Twelve-Factor App, a well-known set of rules for cloud apps. It says an app should not write or manage log files. Each process writes its event stream, unbuffered, to standard output, and the environment routes it.

The time is UTC, the world's reference clock, with milliseconds, marked by the Z at the end. A busy service writes many lines in one second, and milliseconds keep them in order. The OWASP Logging Cheat Sheet, from a non-profit security project, also asks you to synchronise time across all servers.

Logs travel in four hops. The app writes to standard output, an agent on each machine reads and ships the lines, a message queue holds them, and indexer workers write them into storage.

The agent on every machine

An agent is a small program that runs on every server. It reads the log files of every container, adds labels such as service name and host, and sends lines onward in batches. Fluent Bit and Fluentd are common open-source agents.

On Kubernetes, the container runtime saves each container's standard output into files under /var/log/pods on the node, by default. The agent reads those files as they grow.

The agent usually runs as a DaemonSet, a Kubernetes setting that runs one copy on every node, or on the nodes you pick. The Kubernetes docs recommend this, because a node-level agent must run on every node. Once a line has been shipped, losing the container no longer loses the line.

One setting matters for short-lived pods. Fluent Bit's tail input, the part that reads files, looks for new log files every 60 seconds by default (refresh_interval). A pod created and deleted between two scans can be missed, so lower it if such pods matter.

The queue in the middle

The team puts Kafka between the agents and the store. Kafka keeps messages in topics, named streams that many writers add to. Many readers can read the same topic independently, and reading a message does not delete it.

Kafka keeps each message for a set time, whether or not anyone has read it. The default is 7 days (retention.ms of 604800000), as of the Apache Kafka 4.3 docs. The team sets its logs topic to 3 days.

If the search store slows down, agents keep sending to Kafka, and indexers catch up when the store recovers. Without the queue, 40 agents would all retry against a struggling store at once.

Indexers and the two destinations

Indexer workers read batches from Kafka, parse and check each line, and write it to two places. One copy goes to a search cluster for fast queries. The other goes as compressed files to object storage, a cheap store for large files, for long keeping.

Here is the whole path. Follow one log line from the payment services at the top to the two stores at the bottom. Notice that the search screen only ever talks to the search cluster.

When the pipeline is slower than the apps, logs pile up somewhere, and you must decide where they wait and what is dropped first. This push back from a slow stage to a faster one is called backpressure.

On the first salary day after launch, payments jump to four times normal between 6 pm and 9 pm (made up). The indexers fall twenty minutes behind, and Kafka absorbs that easily. The harder case is when Kafka itself cannot take writes, and every agent must hold lines.

How Fluent Bit buffers

Fluent Bit groups records into chunks, small batches it buffers until an output sends them. It can buffer in memory only, or on the filesystem. Its docs say memory-only is more prone to data loss, and filesystem mode is less prone.

In filesystem mode, every chunk is written to a file on disk, and some chunks are also kept in memory. With memory-only buffering, an input with mem_buf_limit set pauses when it reaches that limit, until memory frees up.

Each output can also cap its disk queue with storage.total_limit_size. The docs say that when the cap is reached, the oldest chunk in that output's queue is discarded to make room.

In memory-only mode, a paused input is not instant loss, because the runtime keeps writing to the node's log files and the agent catches up later. But the kubelet, the Kubernetes agent on each node, rotates those files at 10 MiB and keeps 5 per container by default, so a long pause loses the oldest lines.

Choosing what to lose first

The disk cap drops by age, not by level. So the team gives WARN, ERROR and payment events one tag, and everything else another, each with its own output and cap. Fluent Bit's rewrite_tag filter does this by matching a field against a pattern. Python writes WARNING and CRITICAL, not WARN, so the team's pattern matches those spellings too.

One trap: that filter re-emits records through its own buffer, which is in memory with a 10M limit by default. Its docs recommend emitter_storage.type set to filesystem when outputs may push back.

What just happened? Neither queue drops anything until its own cap is reached. The low-level queue has the smaller cap and usually far more lines, so it overflows first while the important queue keeps going.

Counting what you drop

Dropping is acceptable only if you count it. A gap you know about is a nuisance. A gap nobody knows about can hide the one line an investigation needs.

Fluent Bit can serve metrics on port 2020 when its HTTP server is on. The counter fluentbit_output_dropped_records_total has one value per output, so separate outputs give the team a count per level.

Here is a detail the docs do not spell out. The monitoring page describes that counter as records that hit an unrecoverable error or ran out of retries. In the source we read for releases 3.0.0, 4.0.0 and 5.1.4, chunks evicted by the disk cap are added too.

The agent also logs a warning that a chunk was evicted from an output queue to make room under storage.total_limit_size. This is undocumented as of 19 October 2026, so pin your version and test it before you alert on it.

# On a node, with http_server on in Fluent Bit's service settings
curl -s 127.0.0.1:2020/api/v1/metrics/prometheus | grep dropped_records

You should see a HELP line and a TYPE line, then one fluentbit_output_dropped_records_total line per output, labelled with its name. In normal running each number stays at 0, and an alert on any rise tells you a gap has opened.

Sampling is another safety valve. It means keeping only a share of noisy lines. The team keeps every WARN and ERROR line, but only one in ten DEBUG lines from the busiest health-check endpoint.

Logs are stored in time-ordered chunks, and an index points from search terms to those chunks. The big design choice is how much you index: every word in every line, or only a few labels.

Full-text index

A search engine such as Elasticsearch builds an inverted index. Elastic's docs describe it as a list of every unique word that appears in any document, with the documents each word occurs in.

Searching any word across billions of lines is then fast. The price is a large index that needs fast disks and plenty of memory.

Label index

A label-based store such as Grafana Loki indexes only a few labels, such as service, environment and level. Loki's docs state that the content of each log line is not indexed. Lines that share the same labels form a stream, stored as compressed chunks.

To search, Loki picks the streams that match your labels and time range, then scans those chunks for your text. That is the thana writer pulling one beat's diaries for one week, and reading only those.

The catch is cardinality, the number of different values a label can take. Loki's docs warn that high cardinality makes it build a huge index and flush thousands of tiny chunks to the object store.

So a request id is never a label. It stays inside the line, where a scan or a full-text index can find it. For high-cardinality values you search often, Loki's docs point to structured metadata instead.

The team chose full-text search for recent logs, because support staff search by UPI reference numbers that appear inside messages. A label store would also work if those numbers were always in a known field.

Data streams and rollover

In Elasticsearch, logs usually go to a data stream, one name over a series of hidden backing indices. Writes go to the newest one. Rollover starts a fresh backing index when the current one reaches a set size, document count or age.

Old data then leaves by whole backing indices. Elastic's shard sizing guide says deleted documents are only flagged at first, while a deleted index's files are removed at once. So deleting a whole index is far cheaper than deleting lines.

Keep logs only as long as someone will really use them: days or weeks for debugging, longer for audits. Every extra day adds storage, so move older logs to cheaper tiers on a fixed schedule.

The storage maths

The team writes about 60 crore (600 million) log lines a day, averaging 400 bytes each. That is 240 GB of raw logs a day.

Spread over 86,400 seconds, that is about 6,900 lines and 2.8 MB a second on average. The evening rush runs at four times that, so the team sizes the pipeline for about 28,000 lines and 11 MB a second.

Kafka keeps 3 days, which is 720 GB per copy. The team keeps three copies of every message, a Kafka setting called the replication factor, so that is about 2.2 TB of Kafka disk.

The hot tier keeps 7 days on fast disks, with one replica, a second copy on another machine. Assume the indexed size is about the raw size, and measure yours. That is 240 GB times 2 copies times 7 days, about 3.4 TB.

The warm tier keeps days 8 to 30 on cheaper disks with no replica, about 5.5 TB. Without a replica, a lost disk removes those days from search, but the archive still has them.

How much does the archive shrink?

Log text compresses well, but random ids do not. We generated 200,000 JSON lines shaped like the team's, with random trace, request and payment ids. Gzip, the common compression tool, made them 4.9 times smaller at its default level. Zstd, a newer tool, made them 6.3 times smaller at level 19, a slow setting near its top level of 22.

So the team plans for five times smaller: 48 GB a day, about 17.5 TB a year. Lines with fewer random ids shrink more. Measure your own ratio on one real hour of logs before you size anything.

wc -c one-hour.log                # raw size in bytes
gzip -c one-hour.log | wc -c      # compressed size in bytes
# ratio = first number / second number

How a day of logs ages

Each backing index moves through these stages on a timer. Start at the write index and follow both paths: one through the search cluster, one through the archive.

This is Elasticsearch's index lifecycle management, or ILM. Its docs describe the warm phase as an index updated rarely or not at all, but still queried. In the delete phase, the index is no longer needed and can safely be removed.

Two details surprise people. If an index has rolled over, its phase ages count from the rollover, not from when the index was created. And ILM checks on a timer, every 10 minutes by default. The docs' own example says a rollover can then take up to 20 minutes to complete.

So one backing index is not one calendar day. Filter searches by the time field, never by index name.

Clean the data inside the app, before it leaves. Otherwise every copy downstream, in the queue, the index and the archive, holds it for months. Then limit who can search what remains.

The OWASP Logging Cheat Sheet lists data that should usually not be logged directly. It includes passwords, session ids, access tokens, payment card holder data and encryption keys. It says such data should be removed, masked, sanitized, hashed or encrypted.

The team learns this the hard way. A developer logs whole request bodies while chasing a bug. Card numbers sit in the search cluster for three days before anyone notices.

The naive filter and its leak

The team's first fix was a logging filter, which sees every record before the handler writes it and can change it. It replaced card-like numbers in record.msg, the message text, and it looked right.

It does not. Python fills in the %s later, from record.args, when the line is formatted. The filter cleaned the template and never saw the number. On Python 3.13.16 the naive filter printed the full card number.

It also missed card numbers written with spaces, and a field named Password, because its check was case-sensitive. Here is the fixed filter, added to the handler from the first code block.

import logging, re

CARD = re.compile(r"\b(?:\d[ -]?){12,18}\d\b")
SECRET_FIELDS = {"password", "otp", "upi_pin", "card_number", "token"}

class RedactFilter(logging.Filter):
    def filter(self, record):
        record.msg = CARD.sub("[CARD]", record.getMessage())
        record.args = None  # the message is already filled in and cleaned
        fields = getattr(record, "fields", {})
        record.fields = {k: "[REDACTED]" if k.lower() in SECRET_FIELDS
                         else CARD.sub("[CARD]", v) if isinstance(v, str) else v
                         for k, v in fields.items()}
        return True  # keep the line, now cleaned

handler.addFilter(RedactFilter())
log.info("charging card %s", "4111 1111 1111 1111", extra={"fields": {"OTP": "482913"}})

The filter calls record.getMessage(), which fills in the arguments first, then cleans the finished text. It clears record.args, so the text is not filled in twice. Text field values are cleaned too, and field names are compared in lower case. Run both blocks together, and the last line printed looks like this.

{"ts": "2026-10-19T09:15:42.371Z", "level": "INFO", "service": "settlement", "request_id": "req_7c41e9", "msg": "charging card [CARD]", "OTP": "[REDACTED]"}

What just happened? The spaced card number became [CARD], and the OTP field became [REDACTED], even with its name in capitals. We ran both code blocks together on Python 3.13.16.

The pattern catches 13 to 19 digits, with single spaces or dashes allowed between them. It also masks other long numbers, such as 13-digit millisecond timestamps. For logs, masking too much is the safer mistake.

Know its limits: it cleans only lines that pass through this handler. It skips numbers stored as integers, nested objects and tracebacks, and it cannot spot a secret with no fixed shape. A filter is a safety net. The real fix is never logging whole request bodies, and checking new log lines in code review.

• Stop log injection, where user input containing a line break forges a fake log line. JSON lines handle this, because json.dumps writes a newline as a backslash and n. Plain-text logs must strip CR and LF, as OWASP advises.
• Encrypt traffic between agents, Kafka and the search cluster.
• Give support staff access to recent logs only, and record every search they run.
• Make the archive write-once, so nobody can quietly change history.

You search by its ids. A payment id returns every line about that payment, across all its requests. A trace id or request id returns one request's path. Either way, the lines come back from every service, sorted by time.

A month later, another merchant reports a pending payment. The engineer types the payment id and the last hour into the search screen. In about two seconds (made up), 23 lines from six services appear in order.

The story is clear at once. The bank callback arrived on time, but the ledger's database write timed out, and the retry job had not run yet. She finds the cause in four minutes, not two hours.

Sorting by time works only if clocks agree. If two servers differ by a second, lines appear out of order. The OpenTelemetry log data model keeps two times: Timestamp, when the event happened, and ObservedTimestamp, when the collector saw it. Neither fixes a wrong clock, so keep every server's clock synchronised.

Walk through it as a pipeline, and attach a number or a trade-off to every stage.

1. 1

   Ask about scale and use

   How many lines a second, how long to keep them, who searches, and what they search for. Payments support and audits need different answers.

2. 2

   Estimate out loud

   Turn lines a day into lines a second, bytes a day and storage per tier. Size for the peak, not the average.

3. 3

   Draw the pipeline

   Structured lines to standard output, one agent per node, Kafka, indexer workers, then a search cluster and an archive.

4. 4

   Handle overload

   Disk buffers with caps, low levels routed to a smaller queue, every drop counted, and logging that never blocks the app.

5. 5

   Choose the storage

   Full-text or label index, rollover by size or age, hot and warm phases, an archive and a delete policy.

6. 6

   Cover safety and cost

   Request and business ids on every line, masking in the app, limited search access, and the monthly storage bill.