How GFS and HDFS Work: Chunks, One Master, Relaxed Rules

When files outgrow one server, you run out of disk, network speed and safety at the same time. No single machine is big, fast or reliable enough, so you need many cheap machines that behave like one large disk. The GFS HDFS design, shared by the Google File System and the Hadoop Distributed File System, is the classic answer.

Picture a common setup. A film archive digitises old films, and each restored film becomes one file of about 400 GB. The setup is made up, but the problem is real.

The archive plans to scan 6,000 films. That is about 2.4 PB, or 2,400 terabytes. Today everything sits on one storage server with 200 TB of disks, and the storage team lists what breaks first.

• Space. The server is already two-thirds full, and the plan needs twelve times its total size.
• Speed. Thirty scanning stations write at once, and the server's one network card is always busy.
• Failure. Last month one disk died, and a whole film had to be scanned again from the old reel.

Buying a bigger server only moves the wall. The team needs a file system spread over hundreds of ordinary machines. This guide follows the archive through both systems.

The GFS HDFS design cuts every file into large fixed-size chunks, stores each chunk on several ordinary machines, and keeps one master that knows which machine holds which chunk.

Four words carry the whole design. A chunk is one fixed-size piece of a file; HDFS calls it a block. A storage server keeps chunks on its local disks. GFS calls it a chunkserver, and HDFS calls it a DataNode.

The master is one machine that holds the metadata. GFS calls it the master, and HDFS calls it the NameNode. A replica is one copy of a chunk, and both systems keep three by default.

Picture a travelling circus moving its big tent to the next town. The canvas is too large for one truck, so the crew cuts it into equal panels and loads them across several trucks.

The manager keeps a list: panel 7 is on truck 3, panel 8 is on truck 5. He never carries canvas himself. Each panel also travels on a second and third truck, so one breakdown never stops the show.

Map it back. The tent is a 400 GB film. The panels are chunks, and the trucks are storage servers. The manager's list is the master's metadata, and the extra panels are replicas.

Where the picture breaks: the manager writes down where every panel goes. A GFS master does not save chunk locations at all. It asks the storage servers what they hold, as you will see below.

Here is the shape of a cluster, meaning the whole group of machines working as one file system. Notice which arrows carry film bytes and which carry only small questions.

The GFS paper states the rule plainly: clients never read or write file data through the master. They ask it where to go, cache the answer for a limited time, and then talk to storage servers directly.

Chunks are large, 64 MB in the GFS paper and 128 MiB by default in HDFS, so the master has fewer pieces to track and fewer questions to answer. A laptop file system uses blocks of a few kilobytes.

MiB means mebibyte, 1,048,576 bytes. The HDFS setting dfs.blocksize defaults to 134,217,728 bytes, which is exactly 128 MiB. The HDFS design guide simply calls 128 MB a typical block size.

The GFS paper gives three reasons for big chunks. A client needs only one location lookup per chunk. It can keep one network connection open to a storage server for a long time. And the master's metadata stays small.

Try the maths for one 400 GB film. With 128 MiB blocks, that is 2,981 blocks, the last one only partly full. With 4 KB blocks, the same film would need about 100 million entries.

A partly full last chunk does not waste a whole chunk of disk in GFS. The paper says lazy space allocation, growing the chunk's file only as data arrives, avoids that waste.

Big chunks have a cost too. A small file fits in one chunk, so every reader of that file hits the same few machines. The paper calls these overloaded machines hot spots.

It describes a real case. A program was stored as a single-chunk file and then started on hundreds of machines at once, which overloaded the few servers holding it. Google fixed it with more replicas and staggered start times.

You can check the block size and copy count on any HDFS cluster. Run hdfs getconf -confKey dfs.blocksize; on a default setup it prints 134217728, in bytes.

For one file, run hdfs dfs -stat "%o %r" with the file's path. The %o part prints the block size in bytes and %r prints the replication factor, so a default file shows 134217728 3.

Turning a byte position into a chunk

Every client must turn a byte position into a chunk number before it asks the master anything. Here is that step in Python, with a stand-in master so the code runs on its own.

CHUNK_SIZE = 64 * 1024 * 1024    # the GFS paper's 64 MB, as 64 MiB

def locate(file_name, offset, master, cache):
    index = offset // CHUNK_SIZE              # which chunk holds this byte
    key = (file_name, index)
    if key not in cache:                      # ask the master only on a miss
        cache[key] = master.lookup(file_name, index)
    handle, replicas = cache[key]
    return handle, replicas, offset % CHUNK_SIZE

class StandInMaster:                          # pretend master for this demo
    calls = 0
    def lookup(self, file_name, index):
        self.calls += 1
        return 1000 + index, ["server-1", "server-4", "server-7"]

master, cache = StandInMaster(), {}
print(locate("film-0001.mkv", 200_000_000, master, cache))
for offset in range(0, 400_000_000, 1_000_000):   # 400 reads of 1 MB each
    locate("film-0001.mkv", offset, master, cache)
print("master lookups:", master.calls)

Run it with Python 3 and you get two lines. The first is (1002, ['server-1', 'server-4', 'server-7'], 65782272). The second is master lookups: 6.

What just happened? Byte 200,000,000 falls in chunk 2, because chunk numbers start at 0. The last value is the position inside that chunk. Then 400 reads touched chunks 0 to 5, and the cache meant only six questions reached the master.

The handle is the chunk's ID. In GFS it is an unchangeable, globally unique 64-bit number that the master assigns when it creates the chunk. The 1000 + index in the demo is only a stand-in.

The master keeps all metadata in memory: file and directory names, which chunks make up each file, and where each chunk's replicas live. It stores no file data at all.

Memory makes the master fast. It answers a lookup without touching disk, and it scans its whole state in the background. That scan finds chunks that lost a copy and need re-replication, which means making a new copy.

Memory is also its limit, so check the numbers. The GFS paper says the master keeps less than 64 bytes for each 64 MB chunk. The archive's 2.4 PB is about 36 million chunks of 64 MiB, so under 2.3 GB of chunk metadata.

That fits easily on one server. Many small files are the real danger, and the engineer section shows the maths.

How the master survives a restart

Memory is lost when a machine restarts, so names and the file-to-chunk mapping also go to disk. GFS appends each change to an operation log, a file that records every metadata change in order.

GFS copies that log to other machines. It answers a client only after the log record is flushed to disk both locally and on those remote machines. So an acknowledged change survives a crash.

Replaying a long log is slow. So the master saves a checkpoint, a full snapshot of its state, from time to time. Recovery loads the newest checkpoint and replays only the log written after it.

HDFS uses the same pattern with different names. The log is the EditLog, and the snapshot is the FsImage. In Hadoop 3.5.0, a checkpoint runs every hour or after 1,000,000 new transactions by default, whichever comes first.

Chunk locations are asked for, not saved

The GFS master does not save chunk locations at all. It asks every storage server what it holds at startup, then stays current through heartbeats, small regular I-am-alive messages.

HDFS works the same way. Each DataNode sends a heartbeat every 3 seconds by default. It also sends a Blockreport, a list of every block it holds, at startup and then every 6 hours by default.

Between full reports, a DataNode tells the NameNode each time it receives or deletes a block. So the NameNode does not wait six hours to learn about a new block. The NameNode's answers to heartbeats carry orders, such as copy this block to another DataNode.

A write reaches all three replicas in two phases. First the client pushes the bytes along a chain of storage servers. Then one replica, the primary, tells all of them when and in what order to apply it.

The primary is chosen by a lease, a time-limited permission the master grants to one replica of a chunk. In GFS a lease starts at 60 seconds, and the primary can keep extending it while writes continue.

Follow the boxes in order. Each box is a single message, and the detail line says who talks to whom.

The chain matters for speed. Each server forwards data to the closest server that does not have it yet, and starts forwarding as soon as bytes arrive. That way each machine's full outgoing bandwidth is used.

HDFS writes through a similar pipeline. The NameNode picks the DataNodes. The first one receives data in portions, writes each portion locally and passes it to the second, which passes it to the third.

Where the three copies go

Copies are spread across racks. A rack is one cabinet of servers that shares a power supply and a network switch, so one fault can take out the whole rack.

HDFS's default placement for three copies, from its design guide, goes like this.

1. 1

   First copy

   On the writer's own machine if that machine is a DataNode. Otherwise on a random DataNode in the writer's rack.

2. 2

   Second copy

   On a DataNode in a different, remote rack, so losing a whole rack still leaves a copy.

3. 3

   Third copy

   On a different DataNode in that same remote rack. This saves traffic between racks, which is usually slower and busier.

The GFS paper sets the same goal in one line: spread chunk replicas across racks.

Relaxed consistency means GFS does not promise that every replica holds identical, tidy bytes after every write. It promises less and asks applications to handle the gaps, in exchange for speed and simplicity.

The paper uses two words. A file region is consistent when every client sees the same data, whichever replica it reads. It is defined when it is consistent and clients see exactly what one write put there.

• One write succeeds with no other writer: the region is defined.
• Several writers succeed at the same time: the region is consistent but undefined. Everyone sees the same bytes, which may be mixed pieces from several writes.
• A write fails on some replicas: the region is inconsistent. Different clients may see different data.

Record append and duplicates

GFS also offers record append. The client hands over only the data, and GFS picks the position. Many machines can append to one shared file without any locking between them.

Pause and predict. An append succeeds on two replicas and fails on the third, so the client retries the whole record. What does each replica hold afterwards?

The retry lands at a new position on all three. So two replicas now hold the record twice, and the third holds a partial first try or nothing useful in that spot. GFS promises each record lands at least once, as one unbroken piece.

A record can also hit the end of a chunk. If it would not fit in the 64 MB chunk, the primary pads the rest of the chunk with filler and tells the client to retry on the next chunk.

To limit that waste, one record can be at most a quarter of the chunk size, so 16 MB. The paper says this keeps worst-case fragmentation at an acceptable level.

Now imagine the thirty scanning stations appending catalogue entries to one shared log file. One entry might say that reel 14 of a 1962 film is scanned. The indexer that reads this log must skip padding, broken pieces and duplicates.

The GFS paper describes the fix. Each record carries a checksum, a short number computed from its bytes that changes if any byte changes. Each record also carries a unique ID. Here is a reader that uses both.

import json
import zlib

def make_record(entry):
    body = json.dumps(entry).encode()
    return body + b"|" + str(zlib.crc32(body)).encode()

def read_catalogue(raw_records):
    seen = set()
    for raw in raw_records:
        body, _, crc = raw.rpartition(b"|")
        if not crc.isdigit() or zlib.crc32(body) != int(crc):
            continue                  # padding, fragment or corrupt record
        entry = json.loads(body)
        if entry["id"] in seen:
            continue                  # duplicate from a retried append
        seen.add(entry["id"])
        yield entry

good = make_record({"id": "s07-0001", "reel": 14, "status": "scanned"})
other = make_record({"id": "s12-0002", "reel": 3, "status": "scanned"})
raw = [good, b"\x00" * 16, good, other[:20], other]
print([e["id"] for e in read_catalogue(raw)])

Run it and it prints ['s07-0001', 's12-0002']. Five raw records went in: one good record, 16 bytes of filler, a duplicate, a broken fragment and a second good record. Two came out.

What just happened? The checksum test threw out the filler and the fragment, because neither ends in a correct checksum. The seen set threw out the retried copy by its ID.

One honest limit: this demo assumes you can already split the log into candidate records, for example one per line. A real format needs record boundaries, such as a length prefix, that a reader can find again after junk.

HDFS took a stricter path. Its design guide says files are write-once, except for appends and truncates, and have strictly one writer at any time. So HDFS avoids the many-writers mess, but thirty stations cannot share one log file.

On HDFS, the archive gives each station its own log file and lets the indexer read all thirty. It still keeps the checksum and ID in every record, because a retry can happen in any program.

When a storage server fails, the master notices that its heartbeats stopped and has surviving replicas copy its chunks to other servers. Readers simply use another copy meanwhile.

Back to the circus. A truck breaks down on the highway. The manager checks his list, sees which panels it carried, and tells other trucks to make spare copies of those panels. The show still opens on time.

HDFS marks a DataNode dead only after a long silence, over 10 minutes by default. The design guide calls this conservatively long, so a server that flaps up and down does not set off a storm of copying.

Sometimes a server comes back with old data, because it was down during a write. GFS catches this with a chunk version number. The master raises the number each time it grants a new lease.

A replica with an old number is stale. The master treats it as if it does not exist when it answers clients, and removes it during its regular garbage collection, a background clean-up of unused chunks.

Disks can also corrupt data silently. GFS splits each chunk into 64 KB blocks, each with a 32-bit checksum. A storage server checks those checksums before it returns any data.

HDFS checks too. By default it keeps one checksum for every 512 bytes of a block. A client that finds a mismatch can read that block from another DataNode that holds a copy.

GFS handles a dead master differently from HDFS. If the master process crashes, the paper says it can restart almost instantly. If its machine dies, monitoring outside GFS starts a new master elsewhere from the copied operation log.

Meanwhile, read-only shadow masters keep serving reads. The paper says they may lag the real master slightly, typically by fractions of a second.

GFS and HDFS share the same shape, but HDFS is open source, uses bigger default blocks and allows one writer per file. Since Hadoop 2.0.0 it also supports optional standby NameNodes for failover. The table puts them side by side.

The HDFS design guide adds one more idea: moving computation is cheaper than moving data. Jobs run on the machines that already hold the blocks. That is why HDFS and batch tools like MapReduce grew up together.

An HDFS cluster set up for high availability (HA) runs a standby NameNode that replays every edit from a shared, replicated log. It can then take over without a slow cold restart. Before Hadoop 2.0.0, the NameNode was a single point of failure: one part whose crash stopped everything. A default cluster still has one NameNode, so HA is something you configure.

The setup that the HDFS documentation recommends is the Quorum Journal Manager. It uses a small group of JournalNodes, machines that store the edit log. ZooKeeper is a small coordination service, and a ZKFC (ZKFailoverController) process beside each NameNode uses it to pick the active one. In the diagram, follow the edits from the active NameNode to the standby.

Four details make this work. An edit counts only after a majority of JournalNodes store it. With N JournalNodes the log survives (N - 1) / 2 failures, so three tolerate one and five tolerate two.

Second, DataNodes send heartbeats and block reports to every NameNode, so the standby already knows where blocks are. Third, the standby must read every edit from the JournalNodes before it promotes itself.

Fourth, the JournalNodes accept only one writer at a time. That prevents split brain, where two NameNodes both believe they are in charge and write conflicting edits.

Each ZKFC health-checks its NameNode and holds a ZooKeeper session while that NameNode is healthy. The one that wins a lock in ZooKeeper runs the failover. Automatic failover is off by default in Hadoop 3.5.0; you turn it on by setting dfs.ha.automatic-failover.enabled to true and listing your ZooKeeper servers in ha.zookeeper.quorum.

Here is the catch engineers miss. The documentation still says it is desirable to configure fencing methods even with the Quorum Journal Manager. Fencing means cutting off the old active NameNode.

Why bother, if the JournalNodes already block a second writer? The documentation explains that the old active may still serve reads that are out of date. It stops only when it next tries to write to the JournalNodes, and fencing closes that window.

The documentation also says the standby performs checkpoints, so an HA cluster needs no Secondary NameNode. It allows more than two NameNodes, recommends three and suggests not going past five.

With default settings, HDFS declares a DataNode dead after 630 seconds of silence, which is 10 minutes 30 seconds. The design guide only says over 10 minutes, so the exact number comes from the source code.

In the Hadoop 3.5.0 source, the timeout is twice the recheck interval plus ten heartbeat intervals. The recheck interval defaults to 300,000 milliseconds and the heartbeat to 3 seconds.

So the sum is 2 × 300 seconds plus 10 × 3 seconds, which is 630 seconds. Change either setting and the timeout moves with it. As of 19 October 2026 the design guide does not give this formula, so treat it as source behaviour, pin your version and check again on upgrade.

The design guide keeps this timer long on purpose, to avoid a storm of copying when servers flap. The cost: if two of a block's three servers go silent together, that block lives on one copy for over ten minutes.

To see cluster statistics for your DataNodes, run hdfs dfsadmin -report. To see the blocks of one file and where they live, run hdfs fsck, the file system check, with the file's path and the options -files -blocks -locations.

The -locations option prints the DataNodes that hold every block. On a healthy file with default settings, you should see three locations for each block.

GFS and HDFS struggle with millions of small files, with fast random reads and writes, and with the storage cost of three full copies. Each limit comes from a choice that helps huge files read in long streams.

Millions of small files

The archive wants a 50 KB thumbnail for every two seconds of film. That is about 20 million small files, and each one costs NameNode memory, however tiny it is.

A Cloudera engineering post from 2009 gives a rule of thumb: every file, directory and block is an object of about 150 bytes in NameNode memory. Treat it as a rough guide, not a measurement of your version.

By that rule, the 6,000 films with their 18 million blocks need about 2.7 GB. The 20 million thumbnails add a file object and a block object each: 40 million objects, or about 6 GB more.

So the thumbnails would cost more NameNode memory than the films themselves, while using a tiny share of disk. The fix is to pack the thumbnails into one large container file per film, with an index of where each one starts.

With one container file and one index file per film, the NameNode tracks 12,000 extra files instead of 20 million. Hadoop's own options for this include SequenceFiles, which store many small records as key-value pairs in one file.

Fast random reads

These systems are built for streaming through huge files, not for answering one tiny lookup in a few milliseconds. That kind of access belongs to a table store built on top, such as Bigtable.

The cost of three copies

Three-way replication costs 200% extra storage, as the HDFS documentation notes. Erasure coding stores parity pieces instead of full copies. These are extra pieces calculated from the data, enough to rebuild lost ones.

The RS(6,3) policy uses Reed-Solomon, a standard error-correcting code, with 6 data pieces plus 3 parity pieces. Six blocks of data then use nine blocks of disk instead of eighteen, a 50% overhead. HDFS enables RS(6,3) by default, but no file uses it until you set it on a directory with hdfs ec -setPolicy. Only files created there afterwards are erasure coded.

RS(6,3) can lose any 3 of the 9 pieces and still rebuild the data. Our toy test of the same 6-of-9 maths rebuilt the data from every one of the 84 possible six-piece subsets. Three-way replication survives only 2 lost copies.

The price is real. The documentation says erasure coding adds CPU and network load, and that most reads and writes of erasure-coded files go off-rack, meaning they cross between racks. For rack fault tolerance, RS(6,3) needs at least 3 racks, and ideally 9 or more.

In Hadoop 3.5.0, truncate on an erasure-coded file fails with an error. The hflush and hsync calls normally make unfinished data visible to new readers or push it to disk. On erasure-coded files they silently do nothing, so they give neither promise. Append works only in a special mode that starts a new block.

The documentation recommends erasure coding for warm and cold data that is rarely read or written. That fits finished film masters, which are written once and read a few times a year. It does not fit films still being restored, which are being written right now.

The single master, again

The deepest limit is the single master's memory and CPU. Google says it built Colossus, the successor to GFS, to solve metadata scaling limits it hit with GFS.

Colossus spreads metadata across many servers called curators, and stores that metadata in Bigtable. Google's 2021 blog post says this let Colossus scale over 100 times beyond the largest GFS clusters.

Explain it as a split between one metadata brain and many simple storage boxes, then defend each choice with a fact about the workload.

1. 1

   State the workload

   Huge files, mostly written once and read in long streams, on cheap machines that fail often.

2. 2

   Split files into big chunks

   Give the size, 64 MB in GFS or 128 MiB in HDFS, and say it keeps metadata small and lookups rare.

3. 3

   Separate metadata from data

   One master holds names and chunk locations in memory. Clients move bytes directly with storage servers.

4. 4

   Replicate across racks

   Three copies, with at least one on another rack. Heartbeats find dead servers, and the master orders new copies.

5. 5

   Explain the write path

   A lease picks a primary, data flows along a chain, and the primary orders every write.

6. 6

   Name the consistency trade-off

   GFS allows duplicates and padding on append, so readers use checksums and IDs. HDFS keeps one writer per file instead.

7. 7

   Cover the master's failure

   Replicated edit log, checkpoints, an optional standby with automatic failover once enabled, and fencing for the old active. Mention that Colossus later spread metadata across many servers.