MapReduce vs Spark: Disk, Memory and What RDDs Changed

On MapReduce, every pass over the data is a separate job that reads everything from disk and writes its result back to disk. A job that needs 20 passes pays that disk cost 20 times.

Picture a common setup. An agriculture analytics team keeps 18 years of plot-level farm records: district, year, crop, rainfall, harvest in tonnes and sown area in hectares. The setup and its numbers are made up, but the problem is real.

The records come to about 4 TB on a cluster of 20 machines, each with 64 GB of memory. A cluster is a group of machines that work together on one job.

An analyst on the team builds a model that predicts wheat yield, meaning tonnes of grain per hectare. The model learns in rounds. Each round reads every wheat record, measures how wrong the current guess is, and nudges the guess a little.

It needs about 20 rounds. On Hadoop MapReduce each round took about 9 minutes, almost all of it rereading the same records. Twenty rounds of 9 minutes is 180 minutes, so the model took three hours.

Then the analyst started asking quick questions, each one a new job of about 6 minutes:

• What was the average wheat yield in Vidisha in years with under 700 mm of rain?
• Which ten districts lost the most yield in the 2019 dry spell?
• Does the answer change if irrigated land is left out?

The team moved the work to Apache Spark and kept the parsed wheat rows in memory, about 200 GB in total. The model fell to about 23 minutes, and repeat questions came back in seconds. This guide explains why, and where that trick stops working.

You need five words: distributed file system, partition, job, shuffle and iterative. Each one below has a plain meaning and a link to the team's work.

• Distributed file system: storage that spreads files across many machines and keeps extra copies, so one dead disk loses nothing. HDFS, the Hadoop Distributed File System, is the common one. The team's 4 TB lives in HDFS.
• Partition: one slice of a big dataset, small enough for one machine to handle. The records are split into thousands of partitions, so many machines work on them at once.
• Job: one complete piece of work sent to the cluster, from reading input to writing output. On MapReduce, one round of the yield model is one job.
• Shuffle: moving data between machines so that all records with the same key end up together. To total wheat by district, every Vidisha record must reach the same machine.
• Iterative: repeating the same steps over the same data many times, improving the answer each round. The 20-round model is iterative. A one-time yearly report is not.

MapReduce is a way to process huge data on many machines with two functions you write. Map turns each record into key and value pairs, and reduce combines all values that share a key.

Jeffrey Dean and Sanjay Ghemawat described it in a 2004 paper from Google. You write the two small functions, and the framework handles splitting the input, moving data, and rerunning work on failed machines. Hadoop MapReduce is the best-known open-source version.

Here is the idea in plain Python on one machine. map_record turns one line into a key (district and crop) and a value (tonnes and hectares). reduce_yield gets every value for one key and works out tonnes per hectare.

from collections import defaultdict

def map_record(line):
    district, year, crop, rain_mm, tonnes, hectares = line.split(",")
    yield (district, crop), (float(tonnes), float(hectares))

def reduce_yield(key, values):
    total_tonnes = sum(t for t, h in values)
    total_hectares = sum(h for t, h in values)
    return key, round(total_tonnes / total_hectares, 2)

def run_on_one_machine(lines):
    groups = defaultdict(list)            # stands in for the shuffle
    for line in lines:
        for key, value in map_record(line):
            groups[key].append(value)
    return [reduce_yield(k, v) for k, v in groups.items()]

lines = ["Vidisha,2019,wheat,640,9100,2600",
         "Vidisha,2020,wheat,910,11500,2500",
         "Sehore,2019,wheat,700,7200,2400"]
print(run_on_one_machine(lines))

Run it with Python 3 and it prints [(('Vidisha', 'wheat'), 4.04), (('Sehore', 'wheat'), 3.0)]. Vidisha's two rows give 20,600 tonnes over 5,100 hectares, which is 4.04 tonnes per hectare.

What just happened? The groups dictionary collected every value for each key before reduce ran. On one machine it is just a dictionary. On a cluster it becomes the shuffle, and that is where much of the cost hides.

On a real cluster, one job runs through these steps. The details come from the 2004 paper and the Hadoop MapReduce Tutorial.

1. 1

   Split the input

   The input files are cut into splits, one per map task. The Hadoop tutorial says the HDFS block size, the fixed chunk size HDFS stores files in, is treated as an upper bound for a split.

2. 2

   Run the map tasks

   Each map task reads its split and calls your map function on every record. The scheduler tries to run it on a machine that already holds a copy of that data.

3. 3

   Write map output to local disk

   Map output is buffered in memory, then written to the local disk of the map machine. It is split into one region per reduce task, using a partitioning function.

4. 4

   Shuffle and sort

   Each reduce task fetches its region from every map machine over the network. It sorts what it receives by key, so all values for one key sit together.

5. 5

   Run the reduce tasks

   Each reduce task calls your reduce function once per key. It writes the final output to the distributed file system, where it is stored with copies.

MapReduce writes to disk on purpose, because disk is how it survives failures. Map output goes to local disk, and final output goes to the distributed file system. One job hands data to the next only through those files.

The 2004 paper spells out the trade. When a machine fails, its completed map tasks are run again, because their output sat on its local disk. Completed reduce tasks are safe, because their output is in the shared file system.

That design is sound for one big pass over the data. The trouble starts when a job feeds the next job. The paper notes that users often pass one job's output files as input to another MapReduce call. Round 2 of the model must read what round 1 wrote.

Here is one picture for the rest of the guide. Workers on a coconut plantation sort and count nuts by size, round after round, for different buyers. The MapReduce way is to carry every pile back to the shed after each round.

Where the picture breaks: the field has no size limit, but each machine's memory does. Spark also uses the shed during a shuffle, as a later section shows.

Follow one round of the model in the diagram. Notice the two disk boxes inside a single round, and the arrow that sends the next round back to the start.

The 2012 RDD paper from UC Berkeley names this exact cost. In most frameworks of the time, reusing data between two MapReduce jobs meant writing it to external stable storage, such as a distributed file system. That brought data replication, disk reads and writes, and serialization, which means turning objects into bytes and back.

Apache Spark is a cluster computing engine that can keep data you reuse in memory across many steps and jobs. Its original building block is the RDD, a read-only collection split into partitions across machines.

Matei Zaharia and colleagues at UC Berkeley presented RDDs at the NSDI conference in 2012. They aimed at two kinds of work that existing frameworks handled badly: iterative algorithms, and interactive data mining, where a person asks many questions of the same data.

Back at the plantation, Spark leaves the sorted piles in the field. Each new count walks straight to them, and nothing goes back to the shed unless it has to.

Three ideas make an RDD work. Each one is simple on its own.

• Read-only. You never change an RDD. You make a new one from it, such as the wheat rows from all rows. That makes it safe to rebuild a piece from scratch.
• Lazy transformations. A transformation, like filter or map, only records what to do. Spark does no work when you define it.
• Actions. An action, like count or mean, asks for a real answer. Only then does Spark plan the work and run it on the cluster.

The fourth piece made the difference: cache. Calling cache on an RDD tells Spark to keep its partitions in memory once they are first computed. Later actions read those partitions from memory instead of rebuilding them from the files.

Here is one of the analyst's questions in PySpark, Spark's Python interface. It finds the average wheat yield in dry years for ten rainfall cut-offs. The parsed counter is an accumulator, a shared counter that tasks add to, so you can see how often parse runs. The docs warn it can overcount if Spark retries a task.

from pyspark import SparkContext

sc = SparkContext("local[2]", "dry-year-yield")
sc.setLogLevel("ERROR")
parsed = sc.accumulator(0)                  # counts how often parse runs

def parse(line):
    parsed.add(1)
    district, year, crop, rain_mm, tonnes, hectares = line.split(",")
    return float(rain_mm), float(tonnes) / float(hectares)

wheat = (sc.textFile("records/*.csv")
           .filter(lambda line: line.split(",")[2] == "wheat")
           .map(parse)
           .cache())                        # keep in memory after first use

for limit in range(500, 1500, 100):         # ten rainfall cut-offs
    dry = wheat.filter(lambda r: r[0] < limit)
    print(limit, round(dry.map(lambda r: r[1]).mean(), 2))

print("parse calls:", parsed.value)
print(wheat.getStorageLevel())

We ran this on PySpark 4.2.0 against a made-up sample of 3,000 rows, 1,000 of them wheat. It printed ten lines, from 500 2.22 to 1400 2.7, then parse calls: 1000. With your own records the averages will differ, but parse calls should equal your number of wheat rows if the cache fits. The last line, Memory Serialized 1x Replicated, is the cache's storage level, explained near the end.

It ran 10,000 times: 1,000 rows times ten actions. Without cache, every mean rereads the files and reparses every wheat row. With cache, only the first mean does, as long as all the partitions fit in memory.

Nothing runs when wheat is defined, because those calls are lazy. The first mean is the action that starts real work. local[2] runs Spark on your own computer with two threads.

On a real cluster, change the first line to sc = SparkContext(appName="dry-year-yield") and launch the script with spark-submit, which supplies the cluster address. The records path must then be on HDFS or other storage that every worker can read.

The diagram shows the same pattern for the 20-round model. Follow the arrow out of the question box. It returns to the memory box, not to HDFS.

The main difference is where data lives between steps. MapReduce writes it to disk after every job, while Spark can keep it in memory as a cached RDD. Most other differences follow from that one choice.

The programming model differs too. A MapReduce job has one map phase and at most one reduce phase, so a longer pipeline becomes a chain of jobs. Spark lets you write many steps in one program and plans them together as one graph of work.

The Berkeley paper measured the same pattern on real clusters. It reports Spark up to 20 times faster than Hadoop on iterative machine learning and graph jobs. It also reports querying a 1 TB dataset interactively in 5 to 7 seconds.

Read those numbers with care. They compare against Hadoop 0.20.2, on Amazon EC2 machines, rented cloud servers, with 4 cores and 15 GB of memory, in the paper's own tests. The paper credits the speedup to avoiding disk reads and deserialization by keeping data in memory as Java objects.

Spark rebuilds lost data from lineage, a record of the steps that built each RDD from data on stable storage. When a machine dies, Spark reruns those steps for only the lost partitions, instead of keeping spare copies in advance.

The paper's reason is cost. Copying a working set to other machines sends it over the network, whose bandwidth is far lower than memory's. Because RDDs are built only by coarse-grained steps, applied the same way to every row, the recipe stays tiny.

On the plantation, a shower ruins one worker's sorted pile. The estate keeps no spare copy of every pile. The supervisor's note says which heap in the shed the pile came from, so someone re-sorts just that heap.

In the diagram, the first three boxes are the lineage of the wheat RDD. Follow the arrows from the failed machine to see what Spark rebuilds, and notice what it leaves alone.

The paper tested this on a 75-node cluster running k-means, a method that sorts points into groups over many rounds. Rounds took about 58 seconds. With one machine killed, that round took 80 seconds, then later rounds went back to 58.

The paper adds that the lineage graphs in its examples were all under 10 KB. Replicating the k-means working set instead would have meant copying 100 GB.

We saw the same behaviour on PySpark 4.2.0. We cached 100,000 records in 20 partitions on two executors, the worker processes that run tasks and hold the cache. We used Spark's local-cluster test mode on one computer. Then we killed one executor.

The next count reran parse only for the partitions that executor held. Across our runs that was 9 to 13 of the 20 partitions, or 45% to 65% of the cache. The count after that reran nothing, and the surviving executor's partitions were never recomputed.

Narrow and wide dependencies decide the cost

Lineage is cheap to replay only for some steps. In the paper's terms, a narrow dependency means each parent partition feeds at most one child partition, as with filter or map. Rebuilding one lost piece touches only its own inputs.

A wide dependency means one parent partition feeds many child partitions, as with grouping by district. That step needs a shuffle. Losing its output could mean recomputing pieces from every machine.

Spark plans work around this split. It runs chains of narrow steps together in one stage, and starts a new stage at each shuffle. The paper suggests checkpointing, saving an RDD to stable storage, for long lineage graphs with wide dependencies.

You can see the stage split yourself. Call toDebugString on an RDD built with reduceByKey, Spark's step that combines values per key and so needs a shuffle. The line marked +- is the shuffle boundary, and the indented lines below it belong to the earlier stage.

No. Spark is much faster when the same data is reused across steps. The gap shrinks for a single pass over huge data, for shuffle-heavy jobs, and when the data you cache does not fit in memory.

The team saw this on its yearly archive job. Once a year it reads all 18 years of records, cleans them and writes them out in a new format. Each record is read once and written once, so there is nothing worth caching.

On Spark that job ran only a little faster, by the team's made-up timings. Most of the time went on reading and writing data, which both engines must do.

Shuffles still write to disk

The Spark docs call the shuffle expensive, because it involves disk reads and writes, serialization and network transfers. Map-side results stay in memory until they no longer fit, then are sorted by target partition and written to a single file.

Those files have an upside. The docs say Spark automatically keeps some shuffle data, so a node failure during the shuffle does not force recomputing the entire input. In our test, a second count after reduceByKey reran zero map calls, even without cache.

That reuse covers only the post-shuffle result. Counting the pre-shuffle RDD again reparsed all 3,000 rows, so you still need cache for anything you reuse before a shuffle.

When the cache does not fit

With the default RDD level, MEMORY_ONLY, partitions that do not fit are simply not cached. Spark recomputes them on the fly each time they are needed. MEMORY_AND_DISK instead writes those partitions to disk and reads them back.

We tested both on PySpark 4.2.0 with a deliberately tiny memory budget. We cached 40,000 records of about 2 KB each, in 40 partitions, and counted three times.

Here MB means decimal megabytes, as we printed them. Neither run failed. MEMORY_ONLY quietly did 85% of the work again on every count, which is how a cached job can look slow for no visible reason.

The Berkeley paper scopes its memory claim to scan-based jobs. For those, it says performance degrades gracefully when memory runs short, and partitions kept on disk should perform about as well as existing data-parallel systems.

For most new Spark code, use DataFrames rather than raw RDDs. The Spark Quick Start says that after Spark 2.0, Datasets replaced RDDs as the main interface, and it highly recommends switching for better performance.

A DataFrame is a table of rows with named columns, like district, year and rainfall. In Python every Dataset is a DataFrame, because Python has no typed Datasets. Because Spark knows the columns, it can plan the query itself, while an RDD of Python tuples hides that from Spark.

The RDD interface is still supported, and partitions, lazy steps, caching and shuffles still decide the cost. Two details changed, though, and both were true on Spark 4.2.0 as of 19 October 2026.

• Cache defaults differ. In our test, RDD.cache() used MEMORY_ONLY, while DataFrame.cache() used MEMORY_AND_DISK_DESER, which spills to disk instead of recomputing.
• Spark Connect, the client-server way to reach a Spark cluster, does not support RDDs or SparkContext. The Spark Connect overview says so for both Python and Scala clients.

So if the team moves its model to DataFrames, a cache that does not fit spills to disk by default rather than being recomputed. If it moves to Spark Connect, the RDD code in this guide has to become DataFrame code first.

One more PySpark detail: cached RDDs print as Memory Serialized. The RDD Programming Guide explains that in Python, stored objects are always serialized with the Pickle library, whatever level you choose.

Choose MapReduce mainly when you already run it, the jobs are single large passes that work well, and nobody needs them changed. For new work that reuses data, Spark is usually the better fit.

• Old jobs that work and rarely change. Rewriting them costs time and adds risk for a small gain.
• Single-pass batch jobs on a cluster with little spare memory, where a Spark cache would not fit anyway.
• Teams whose operations are built around MapReduce, where restarting tasks from files on disk is well understood.

If your real choice is between processing data in big scheduled runs or as it arrives, that is a different question. It is about batch versus stream processing, and this guide treats both engines as batch engines.

Explain it as one design choice, disk or memory between steps, and then show which workloads that choice helps.

1. 1

   Draw one MapReduce job

   Input in HDFS, map tasks, map output on local disk, shuffle, reduce tasks, output back to HDFS. Point at both disk writes.

2. 2

   Show a chain of jobs

   Say that an iterative model becomes many jobs, and each one rereads the data from disk. Give a rough cost, such as 20 rounds times 9 minutes.

3. 3

   Introduce the cached RDD

   Read the data once, keep the parsed partitions in memory, and run every round against them. Mention that transformations are lazy and actions start the work.

4. 4

   Answer the failure question before it is asked

   Explain lineage: Spark rebuilds only the lost partitions from the input files, instead of copying memory to other machines.

5. 5

   Name the limits

   Shuffles still write to disk, and a MEMORY_ONLY cache that does not fit gets recomputed. A single pass gains little. This shows you know where the speedup ends.

6. 6

   Bring it to today

   Say that new code uses DataFrames, whose cache spills to disk by default, and that Spark Connect clients cannot use RDDs at all.