Batch vs Stream Processing Explained: Late Data, Lambda and Kappa

You find out about problems a day late. A nightly report only looks at the data after the day is over, so a fault at noon shows up the next morning. Batch vs stream processing is the choice between computing after the data is complete and computing as it arrives.

Picture a common setup. A rooftop-solar monitoring service watches panels on about 1,200 housing societies and factories. The setup is made up, but the problem is real.

Each site has an inverter, the box that turns the panels' direct current into the alternating current a building uses. Every inverter sends one reading a minute: power right now and energy since the last reading. That is about 1.7 million readings a day.

The data team built the first version. The readings land as files in object storage (cheap storage for large files). At 2 am, a job reads the whole previous day, adds up each site's energy and emails every customer a report by 7 am.

It worked for months. Then a factory's inverter tripped at 11:10 am and stopped producing. The factory ran on grid power all afternoon, and nobody knew until the next morning's report showed a flat line after 11 am.

The factory manager put the lost solar energy at about Rs 18,000 (made up). He asked a fair question: why did a monitoring service not notice for twenty hours?

Customers now want two things. A live dashboard with numbers that are minutes old, not a day old. And an alert within minutes when a site stops producing in daylight. Should the team replace the nightly job, run both, or find a way to need only one?

Batch processing collects data into a finished chunk, such as one full day, and processes the chunk in one run. Stream processing handles each event soon after it happens, and keeps its results updated as new events arrive.

Picture a village weather station during the monsoon. The rain gauge is a simple cylinder that collects rain. At the end of the season, the station clerk adds up every daily reading and writes the season summary. That is batch.

Now a heavy downpour starts at dusk. The headman wants to know if the river will flood. So someone reads the gauge every hour and calls the number out to the village. That is streaming.

Both use the same gauge and the same rain. They differ in when the work happens and how fresh the answer is. A few words will help you follow the rest of this guide.

• Event: one fact that happened at a moment, such as one inverter reading at 11:10 am.
• Bounded data: data with a known end, like all of yesterday's readings. Batch jobs work on bounded data.
• Unbounded data: data that never ends, like the readings still arriving now. Streaming jobs work on unbounded data.
• Latency: the delay between an event happening and its result being ready. The nightly report has a latency of up to about 31 hours, for a reading taken just after midnight.
• Window: a slice of a stream, such as five minutes, that you group together so you can count or add things.
• Throughput: how much data a system can handle per second or per hour.

The table puts the two styles side by side for the solar service.

A batch job runs on a schedule, reads a complete set of input, computes results and writes them out in one go. If it fails, you run it again on the same input.

That rerun property is the quiet strength of batch. Yesterday's files never change. So running the job twice gives the same answer twice, and a bug fix only needs one more run.

Here is the nightly pipeline. Follow it from the inverters to the email, and notice that nothing happens between readings arriving and 2 am.

The core of the job fits in a few lines. Each reading is a small dictionary with a reading_id, a site_id and the kWh (kilowatt-hours, a unit of energy) since the last reading.

from collections import defaultdict


def daily_report(readings):
    seen = set()
    totals = defaultdict(float)
    for r in readings:                    # the whole day, already complete
        if r["reading_id"] in seen:       # the modem sent this one twice
            continue
        seen.add(r["reading_id"])
        totals[r["site_id"]] += r["kwh"]
    return {site: round(kwh, 1) for site, kwh in totals.items()}


day = [
    {"reading_id": "A-1", "site_id": "A", "kwh": 0.5},
    {"reading_id": "A-2", "site_id": "A", "kwh": 0.5},
    {"reading_id": "A-2", "site_id": "A", "kwh": 0.5},   # resent
    {"reading_id": "B-1", "site_id": "B", "kwh": 0.4},
]
print(daily_report(day))
print(daily_report(day) == daily_report(day))   # a rerun gives the same answer

Two lines matter. The loop can trust that the day is complete, because the job only starts after midnight. The seen set drops readings the modem sent twice, which is easy when every reading of the day is in hand.

To check it, save the code as batch_demo.py and run python3 batch_demo.py. You should see {'A': 1.0, 'B': 0.4} and then True. The resent A-2 reading is counted once, and a rerun gives the same totals.

Batch also gets cheap throughput. Reading big files in order is fast, and the machines can shut down once the job ends. Billing, monthly analytics and model training often still run this way.

A streaming job runs all the time, reads events from a log as they arrive, and updates results window by window. Results come out in seconds or minutes instead of the next morning.

The log is usually a system like Apache Kafka. A Kafka topic, a named stream of events, is split into partitions. Each partition is an ordered list of events, and each consumer, a program reading the topic, remembers its offset, its position in that list.

Kafka's documentation is precise about order. Events with the same key, such as one site id, go to the same partition, and a consumer reads each partition in exactly the order it was written. The guarantee is per partition, so events from different partitions can reach a job interleaved.

Back to the weather station. During the downpour, the hourly caller does not wait for the season to end. He reads, calls out a number, and keeps going. A streaming job does the same with each window.

Here is the streaming path the team adds. Look at the two outputs at the end: one feeds the dashboard and one sends alerts.

With this path, the factory's trip would show up when its first all-zero window closes. The manager would get an SMS within about twenty minutes, not a report the next day. The engineer half shows where that number comes from.

Batch is still better when nobody needs the answer quickly, when the work needs the whole dataset at once, or when cost and simplicity matter more than freshness. Many strong systems use streaming for a few outputs and batch for the rest.

These signs point to batch for the solar service:

• Monthly bills, which are read once and must be exactly right.
• Yearly reports that compare every site against every other site.
• Training a model that predicts tomorrow's output from a year of weather and readings.
• One-off questions from the analytics team, such as which panel brand ages best.

These signs point to streaming:

• Alerts, where every minute of delay costs a customer money.
• Live dashboards that customers open many times a day.
• Actions that must follow an event quickly, such as switching a factory to backup power.

The Lambda architecture runs a batch layer and a speed layer over the same events. A serving layer then merges their results when someone asks. The batch layer recomputes past days from all the raw data, and the speed layer covers the hours since the last batch run.

Nathan Marz described the idea in a popular blog post, as Jay Kreps recounts in his 2014 essay. Kreps argues that it caught on because teams had fast stream systems that could not reprocess history. Lambda let a batch layer do the reprocessing.

The team's first plan is exactly this. Keep the nightly job and add the streaming job beside it. In the diagram, follow both arrows out of the topic and see where they meet again.

The trouble shows up within a month. A residents' association calls. On Monday evening the dashboard showed 412 kWh for the day, but Tuesday's morning report says 400 kWh (made up).

The team finds the cause. The batch job drops readings the modem sent twice, as in the code above. The streaming job had no such check, so it counted resent readings twice.

This is the core complaint about Lambda. Kreps wrote that keeping code that must produce the same result in two complex distributed systems is painful. You fix a rule in one place and must remember the other.

The Kappa architecture uses only stream processing, and keeps the event log long enough to replay it. When the logic changes, you run a new version of the streaming job from the start of the kept log.

Kreps proposed it in the same 2014 essay, and only tentatively suggested the name. His recipe has four steps, and it depends on a log that can rewind.

1. 1

   Keep the log

   Retain the raw events in a log such as Kafka for as long as you may need to reprocess. Kreps's example: to reprocess 30 days, keep 30 days.

2. 2

   Start a second job

   Run the new version of the job from the start of the retained data, writing into a brand new output table.

3. 3

   Switch when caught up

   Once the new job reaches the present, point readers to the new table.

4. 4

   Retire the old version

   Stop the old job and delete its old output table.

Kafka's design documentation supports this. It describes keeping messages for a relatively long period, such as a week, instead of deleting them once read. It also says a consumer can rewind to an old offset and read the data again.

In the diagram, notice that there is only one job definition. Version 2 is the same code with the duplicate fix, replaying old readings while version 1 keeps serving.

Daily reports can now be 24-hour windows in the same job. The dashboard and the 7 am email can no longer drift apart through two copies of a rule, because one piece of code produces both.

Kappa has its own costs. Replaying a month of readings for 1,200 sites takes real machine time, and the log must be stored for as long as you might replay it.

So the team also keeps the daily reading files in object storage for long history. For a replay older than 30 days, the same job reads those files as a bounded input.

Choose Kappa when one streaming codebase can produce every result you need and you can replay history in reasonable time. Choose Lambda when batch and streaming truly need different logic or tools.

The line between the two has blurred. The Dataflow Model paper, from 2015, reserves the words batch and streaming for execution engines, not for the logic you write. One pipeline can then run on either.

Modern engines follow that idea. Apache Flink runs one program in batch mode over bounded input or streaming mode over unbounded input. Its documentation says both modes give the same final results on bounded input.

Note the word final. In streaming mode, Flink may send several updates for a result before the last one, while batch mode sends one result at the end.

Apache Spark's Structured Streaming lets you write a stream query the same way as a batch query. The Spark 4.2.0 documentation says it runs as a series of small batch jobs, called micro-batches, by default, with end-to-end latencies as low as 100 milliseconds.

Late events make streaming hard because a window must close at some point, yet a reading for it may still be on its way. Close too early and you miss data. Wait too long and results are slow.

Two clocks matter here. Event time is when the reading was taken, stamped by the inverter. Processing time is when the streaming job sees it. The Flink documentation says event-time results are correct and consistent, even out of order, once all the data has arrived.

For the solar service, the gap is real. An apartment complex loses its mobile signal for half an hour. Its modem stores readings and sends them all at once when the signal returns.

Back at the weather station, a volunteer from a far hamlet walks in the next morning with last night's reading. Do you change the number you already called out? Streaming systems face that choice every minute.

Where the picture breaks: a rain gauge never reports the same rain twice. Modems do resend readings, so a streaming job needs both a rule for late data and a rule for duplicates.

The usual tool for late data is a watermark. A watermark is a marker that says no more events older than a certain time are expected. When the watermark passes a window's end, the window closes and its result goes out.

A small watermark you can run

Here is the idea in plain Python, with the apartment complex as site B. Site A sends every minute on time. Site B's 11:00 to 11:29 readings arrive in one burst after site A's 11:30 reading.

WINDOW = 300          # 5-minute windows, in seconds
ALLOWED_DELAY = 600   # wait up to 10 minutes for late readings
windows = {}          # (site_id, window_start) -> kWh so far
max_seen = 0          # newest event time seen so far
late = []             # readings that missed their window


def save_late(r):
    late.append(r)


def publish(key, kwh):
    site, start = key
    print(f"site {site} window {start // 3600}:{start % 3600 // 60:02d} -> {kwh:.1f} kWh")


def on_reading(r):
    global max_seen
    start = r["event_time"] - r["event_time"] % WINDOW
    if start + WINDOW <= max_seen - ALLOWED_DELAY:
        save_late(r)                      # its window already closed
        return
    key = (r["site_id"], start)
    windows[key] = windows.get(key, 0.0) + r["kwh"]
    max_seen = max(max_seen, r["event_time"])
    watermark = max_seen - ALLOWED_DELAY
    for done in [k for k in windows if k[1] + WINDOW <= watermark]:
        publish(done, windows.pop(done))  # dashboard and alert check


def at(h, m):
    return h * 3600 + m * 60


# Site A sends every minute on time. Site B loses signal at 11:00
# and sends its 11:00 to 11:29 readings in one burst at 11:30.
site_a = [{"site_id": "A", "event_time": at(11, m), "kwh": 0.5} for m in range(40)]
site_b = [{"site_id": "B", "event_time": at(11, m), "kwh": 0.4} for m in range(30)]
for r in site_a[:31] + site_b + site_a[31:]:
    on_reading(r)
print("late readings kept aside:", len(late))

Save it as watermark_demo.py and run python3 watermark_demo.py. Expect five 2.5 kWh windows for site A from 11:00 to 11:20, one 2.0 kWh window for site B at 11:20, then the line late readings kept aside: 20.

What just happened? The watermark trails the newest event time by ten minutes. Site A's 11:30 reading moved it to 11:20, so site B's windows from 11:00 to 11:15 had already closed. Their 20 readings went to save_late instead of being lost.

Notice that the watermark is shared. One site sending on time moved it for every site, which is exactly why the late path matters.

How long does an outage alert really take?

Pause and predict: with five-minute windows and a ten-minute wait, how long after a trip at 11:10 does the alert fire?

The answer is 15 minutes for a trip at 11:10, and up to 19 minutes for a trip at 11:11. The first all-zero window must start at or after the trip minute. Then it waits for its own end plus ten minutes.

So a five-minute alert promise needs a shorter wait, smaller windows, or a separate rule that does not wait for windows at all. The team tells the customer team that a shorter wait means faster alerts but more late readings.

Both engines use watermarks, but they track them differently and treat very late events differently. Know which one you run before you promise anyone a number.

In Flink, each parallel source task usually makes its own watermark. An operator that reads several inputs uses the minimum of its inputs' event times. One example is the step after a keyBy, which regroups events by a key such as site id. So a slow input holds windows back, and its events are not treated as late.

Within one input, though, a fast site still moves the watermark for every site in it, just as in the sketch. Flink drops too-late events by default, but it can send them to a side output, a separate stream you can store.

In Spark Structured Streaming, a query with one input keeps a single watermark: the newest event time seen, minus the delay you set. With several input streams, Spark uses the minimum by default. Spark sets the watermark at the start of each micro-batch, from event times in earlier micro-batches. A reading is judged against that value, not against newer readings in the same micro-batch.

Spark's documentation promises never to drop data that is less late than the delay. Data later than that may or may not be counted. So the guarantee only runs one way.

A Spark 4.2.0 test of the same burst

We ran the sketch's data through PySpark 4.2.0 in local mode with Java 21. We sent the readings as three separate files, so each lands in its own micro-batch and the watermark can move between them. To run it yourself, install it with pip install pyspark==4.2.0. The Spark 4.2.0 docs ask for Java 17, 21 or 25 and Python 3.10 or later.

import json, os, shutil, tempfile
from pyspark.sql import SparkSession, functions as F

spark = (SparkSession.builder.master("local[2]")
         .config("spark.sql.session.timeZone", "UTC")
         .config("spark.sql.shuffle.partitions", "2").getOrCreate())
spark.sparkContext.setLogLevel("ERROR")
base = tempfile.mkdtemp()
inbox = os.path.join(base, "in")
os.makedirs(inbox)


def send(name, rows):  # drop one file of readings into the input folder
    tmp = os.path.join(base, name)
    with open(tmp, "w") as f:
        f.writelines(json.dumps(r) + "\n" for r in rows)
    shutil.move(tmp, os.path.join(inbox, name))


def reading(site, minute, kwh):
    return {"site_id": site, "event_time": f"2026-10-19 11:{minute:02d}:00", "kwh": kwh}


readings = (spark.readStream
            .schema("site_id STRING, event_time TIMESTAMP, kwh DOUBLE").json(inbox))
totals = (readings.withWatermark("event_time", "10 minutes")
          .groupBy(F.window("event_time", "5 minutes"), "site_id")
          .agg(F.round(F.sum("kwh"), 1).alias("kwh")))
query = (totals.writeStream.outputMode("append")
         .format("memory").queryName("totals").start())

send("1.json", [reading("A", m, 0.5) for m in range(31)])    # A, on time to 11:30
query.processAllAvailable()
send("2.json", [reading("B", m, 0.4) for m in range(30)]     # B's late burst
     + [reading("A", m, 0.5) for m in range(31, 40)])
query.processAllAvailable()
send("3.json", [reading("A", 50, 0.5)])                      # moves the watermark on
query.processAllAvailable()

spark.sql("SELECT site_id, date_format(window.start, 'HH:mm') AS start, kwh "
          "FROM totals ORDER BY site_id, start").show()
dropped = sum(p["stateOperators"][0]["numRowsDroppedByWatermark"]
              for p in query.recentProgress if p.get("stateOperators"))
print("late groups dropped:", dropped)
query.stop()

Expect the same windows as the sketch, plus 11:25 to 11:35 for site A and 11:25 for site B, because the 11:50 reading moves the watermark on. Each site A window shows 2.5 kWh and each site B window 2.0 kWh.

The difference is the late burst. Spark counted none of site B's readings from 11:00 to 11:19 and kept no copy of them. Nothing in the results table shows they are gone; only a progress metric hints at it.

The last line printed late groups dropped: 4, not 20. In our runs, with one input file per micro-batch, the numRowsDroppedByWatermark metric counted one row per late site and window, not one per reading. This is undocumented as of 19 October 2026; pin your version.

Both engines also save window state in checkpoints, copies written to durable storage, so a crash does not lose half-built totals. Flink's documentation says its streaming mode uses checkpoints for failure recovery. Spark's says state changes are saved to the checkpoint location you provide.

Explain it as a choice about freshness and correctness, made separately for each output. Then show how late data and reprocessing are handled.

1. 1

   Ask how fresh each output must be

   List the outputs, such as alerts, dashboards and bills. Write the latency each one needs next to it.

2. 2

   Draw the event log first

   Put a replayable log such as Kafka at the centre. Both batch and streaming can read from it.

3. 3

   Pick a style per output

   Use streaming where minutes matter and batch where hours are fine and full data helps.

4. 4

   Handle time and late data

   Window by event time, choose a watermark delay, and say where very late events go.

5. 5

   Plan for reprocessing

   Explain how you fix a bug in history: rerun a batch, or replay the log into a new table.

6. 6

   Name your architecture and its cost

   Say Lambda or Kappa, and state its weak point: two codebases, or long retention and slow replays.