Blob Storage Design: How an S3-Style Store Keeps Files Safe

File storage breaks when files sit on a few big servers and nothing can rebuild them after one server fails. At small scale this works for months. Good blob storage design removes that dependence on luck.

Picture a common setup. A team runs a storage service that many apps use for invoices, product photos, app backups and lecture videos. It holds about 400 TB for 1,200 customers, spread over 900 million files. The setup is made up, but the problem is real.

The first version was simple. Six large servers kept files on local disks, protected by RAID (several disks working as one, so a single disk can fail without losing data). A database row said which server and folder held each file.

Then a disk controller on one server went bad and corrupted two disks in the same RAID group. About 31,000 files were gone, and the latest backup was nine days old. One customer lost a week of lecture videos.

So the real question is what promise the team can make about never losing a file, and what it takes to keep that promise. The rest of this guide builds the design that answers it.

You need six words: blob, bucket, key, chunk, metadata and durability. Each one gets a plain meaning and an example from the team's store.

Blob. A blob, short for binary large object, is any file the store treats as plain bytes. It never looks inside. Example: a 2 GiB lecture video or a 40 KB invoice PDF.

Bucket and key. A bucket is a named container for blobs, usually one per customer or project. A key is the blob's full name inside that bucket, such as videos/week-12/lesson-3.mp4. In the team's store, the slashes are just characters; there are no real folders.

Chunk. A chunk is a fixed-size piece of a blob, stored and copied on its own. The team cuts large blobs into 8 MiB chunks, where a MiB (mebibyte) is 1,048,576 bytes. So a 2 GiB video, which is 2,048 MiB, becomes 256 chunks.

Metadata. Metadata is the data about each blob: its size, owner, checksum, and the list of chunks with where each one lives. A checksum is a short fingerprint computed from the bytes. Without metadata, the chunks are anonymous bytes on anonymous disks.

Durability. Durability is the chance that a stored blob is still there, unchanged, after a year. Availability is different: it is the chance you can read the blob right now. A blob can be durable but briefly unavailable.

Real targets show the gap. As of 19 October 2026, AWS says S3 Standard is designed for 99.999999999 percent durability but 99.99 percent availability over a given year. Losing data is meant to be far rarer than a failed read.

Blob storage design is the work of building a service that stores huge numbers of files by key, keeps them safe through hardware failures, and serves them back fast. The core idea is to keep the bytes and the record of the bytes in separate systems.

Picture a family moving an old Rajasthani haveli to a new town, stone by stone. Every stone gets a painted number. A master register lists each number, what the stone is, and which truck carried it. The stones are your chunks, and the register is your metadata.

Lose one stone and the house can still be rebuilt with care. Lose the register, and you own thousands of numbered stones with no idea how they fit. That is why a blob store guards its metadata even more carefully than its chunks.

Where the picture breaks: a haveli has one copy of each stone, while a blob store keeps spare copies or parity for every chunk. The register is copied too, onto several machines that must agree.

Here is the team's design. Start at the customer's app and follow the arrows. Notice that the API servers talk to everything, but the data nodes never decide anything on their own.

The API servers are stateless, meaning they keep nothing that matters between requests. If one crashes, the load balancer sends the next request elsewhere. All lasting state sits in two places: the metadata service and the data nodes.

The interface stays small. PUT stores a blob under a key, GET reads it, DELETE removes it, and LIST returns keys that start with a prefix such as videos/week-12/. Amazon S3 works the same way: its user guide calls a bucket a container for objects, and a key each object's unique identifier in its bucket.

A blob store splits files into chunks so a big file can be spread across many disks, written in parallel, copied piece by piece and checked piece by piece. One huge file on one disk is slow to write, slow to copy and painful to lose.

Take the 2 GiB lecture video. Sent in one request, a network drop at 1.9 GiB means starting again. If the app uploads in parts instead, it resends only the part that failed, and it can send several parts at once.

Amazon S3 offers exactly this as multipart upload. As of 19 October 2026, its limits page says parts are 5 MiB to 5 GiB, with no minimum for the last part, and up to 10,000 parts per upload. The same page caps one object at 48.8 TiB.

That page also says to consider multipart upload once an object reaches 100 MB. Inside the store, the API server then cuts each blob into its own fixed-size chunks, so the bytes can spread over many data nodes.

Chunk size is a trade-off. Small chunks mean more metadata entries and more requests per file. Large chunks mean slower retries and lumpier disk use. The team picks 8 MiB, which keeps the 2 GiB video to a list of 256 entries.

Here is the splitting step on an API server. store_chunk is a stand-in for the code that sends one chunk to three data nodes. The manifest it builds is the chunk list that later goes into the metadata record.

import hashlib
import io

CHUNK_SIZE = 8 * 1024 * 1024  # 8 MiB per chunk

def store_chunk(chunk_id, data, checksum):
    # Stand-in: a real store sends the bytes to three data nodes.
    return ["zone-a/node-07", "zone-b/node-12", "zone-c/node-03"]

def upload_in_chunks(upload_id, stream):
    manifest = []
    index = 0
    while True:
        data = stream.read(CHUNK_SIZE)
        if not data:
            break
        checksum = hashlib.sha256(data).hexdigest()
        chunk_id = f"{upload_id}-{index}"
        locations = store_chunk(chunk_id, data, checksum)
        manifest.append({"id": chunk_id, "size": len(data),
                         "sha256": checksum, "at": locations})
        index += 1
    return manifest

blob = io.BytesIO(b"x" * (20 * 1024 * 1024))  # a 20 MiB test blob
for entry in upload_in_chunks("up42", blob):
    print(entry["id"], entry["size"], entry["sha256"][:12])

Run it with Python 3 and you should see three lines. The first two read 8388608 bytes with the fingerprint 0c77bc0a0795, and the last reads 4194304 bytes (4 MiB) with baa7a6d36ffa.

What just happened? A 20 MiB blob became two full chunks and one short last chunk. The first two fingerprints match because their bytes match, and any later change to the bytes would change the fingerprint.

You can check a fingerprint yourself. On Linux, sha256sum FILE prints the same 64-character hex string that hashlib.sha256 gives for the same bytes. Saved next to each chunk, that string is what later catches damage.

What about millions of tiny files?

Tiny files have the opposite problem. A 40 KB invoice fits in one chunk, but giving every invoice its own file on disk costs extra disk reads just to find it. So data nodes append many small chunks into large volume files instead.

Facebook's Haystack photo store, described in a 2010 paper, made this choice for that reason. In its older file-per-photo design, reading one image often took more than 10 disk operations, mostly for metadata.

Haystack packs millions of photos into one file of about 100 GB, called a volume. It keeps per-photo metadata small enough to hold in main memory, so a read needs at most one disk operation.

A blob store keeps metadata in its own replicated, strongly consistent database, split across machines by bucket and key. It is the single source of truth for which version of a blob is current and where its pieces live.

Each metadata record in the team's store holds a small set of fields:

• Bucket, key and version id, which together name exactly one stored blob.
• Size, content type and the time it was written.
• A checksum of the whole blob, so a full read can be verified end to end.
• The chunk list: each chunk's id, size, checksum and the data nodes that hold its pieces.
• A state flag, such as live or deleted, plus who owns the blob.

Do the rough sizing out loud. Most blobs are small and have one chunk, so say an average record is about 1 KB. Then 900 million blobs need about 900 GB of metadata.

Big blobs cost more. The 2 GiB video's 256-entry chunk list is about 45 KB as JSON, close to the size of the invoice it sits beside. 900 GB on one machine would make that machine a single point of failure and a bottleneck. Split across machines, it is a normal size for a database.

Sharding means splitting one big table across machines. The team splits by ranges of bucket plus key, not by a hash. Range sharding keeps keys with the same prefix together, so a LIST of videos/week-12/ usually reads one shard instead of all of them.

The price of ranges is hot spots. If one customer writes thousands of keys a second under one prefix, one shard gets all of them. The metadata service watches load and splits a busy range into two shards when it grows too hot.

Each shard keeps three copies on machines in different zones. The copies agree on every change through a consensus protocol such as Raft, which accepts a change only after a majority of copies confirms it. So a shard survives one machine dying without losing a committed record.

A blob store avoids loss by storing extra pieces of every blob in separate failure zones, then repairing missing pieces quickly. No single disk, server, rack or zone failure should remove more pieces than the design can rebuild from.

A failure zone is a group of machines that can fail together, such as one data centre with its own power and network. AWS calls these Availability Zones. As of 19 October 2026, its durability page says S3 Standard and most other classes store objects across at least three of them.

The same page says those classes are designed to keep data through the loss of an entire zone. S3 One Zone-IA is the single-zone exception on that page: it stores data across devices inside a single zone. Treat eleven nines as a target for a mature system, not a number you get for free.

Option one: full copies

Replication stores three full copies of each chunk, one per zone. It is simple, reads are fast from any copy, and repair just copies a healthy chunk. The cost is three bytes of disk for every byte a customer stores.

Option two: erasure coding

Erasure coding cuts a chunk into k data pieces and computes m extra parity pieces from them. With a Reed-Solomon code, a widely used erasure code, any k of the k plus m pieces rebuild the whole chunk.

Facebook's f4 store, described in a 2014 paper, uses Reed-Solomon with 10 data and 4 parity blocks, each on a different rack. That costs 1.4 bytes of disk per byte and survives any 4 lost blocks. The paper says each f4 cell sits in one datacenter and cannot survive losing it.

Pause and predict: what happens if the team stretches 10 plus 4 over three zones? Spread 14 pieces over 3 zones and one zone must hold at least 5, because 14 divided by 3 is more than 4.

Losing that zone removes 5 pieces, but the code survives only 4. So 10 plus 4 protects against racks, not against a whole zone. f4 got datacenter safety from a second layer instead.

At first f4 kept a second copy of each coded volume in another datacenter, for 2.8 effective copies. Later it paired volumes in two regions and stored their XOR, a combined parity block, in a third region. That brought the cost to 2.1 copies.

The team wants something simpler for three zones in one region. To survive a zone, the parity count must be at least the pieces per zone. It picks 9 data plus 6 parity pieces, with exactly 5 pieces in each zone.

Losing a whole zone removes 5 pieces, and 10 survive. One more failed disk elsewhere still leaves 9, which is enough to rebuild. A 10 plus 5 code would also survive the zone at 1.5 bytes per byte, but with no spare.

Erasure coding has real costs too. A healthy read of a chunk now touches 9 servers instead of 1. Rebuilding one lost piece means reading 9 others, which uses far more network than copying a replica. Coding each tiny blob on its own gains little, because each piece carries fixed overhead, so the team codes whole volumes once they fill up.

So the team mixes both, much as f4 left new, busy photos in Haystack. New writes land as three full copies in an open volume. When a volume fills up, it is sealed (no more writes), erasure coded in the background, and its extra copies are freed.

When you upload a file, the API server checks your permission, writes every chunk to data nodes, waits for them all to confirm, and only then commits the metadata record. The commit is the moment the blob exists.

Back at the haveli move, the family has one firm rule. A stone is marked as moved in the register only after it is set in the new wall. A stone still on a truck is not yet part of the house.

Follow one upload of the lecture video in the diagram below. Each box is one message, and the note says who talks to whom. The metadata commit comes last.

1. 1

   Check who is asking

   The API server verifies the request signature, a code computed from the request and the customer's secret key, and confirms the customer may write to this bucket. A failed check ends the request before any bytes move.

2. 2

   Pick places for each chunk

   The placement service returns three healthy data nodes in three different zones, preferring nodes with free space and light load.

3. 3

   Write and verify the chunks

   Each data node writes the chunk to disk, recomputes the checksum, and confirms only if it matches the one the API server sent.

4. 4

   Commit the record

   With every chunk confirmed, the API server writes the metadata record with the full chunk list. This single write makes the new version current.

5. 5

   Reply to the app

   Only after the commit succeeds does the app get 200 OK, the HTTP success code. If anything failed earlier, the app gets an error and can retry.

Committing last gives a clean guarantee. A reader sees either the old version or the new one, never a half-written blob. S3 documents the same promise: updates to a single key are atomic, so a reader gets old or new data, never partial data.

S3 also promises strong read-after-write consistency: a read that starts after a successful PUT returns the new data. Its user guide adds two limits worth copying into your design. If two PUTs to the same key race, the one with the latest timestamp wins, and there is no atomic update across two keys.

As of 19 October 2026, S3 also offers conditional writes. With an If-None-Match header, a PUT fails with 412 Precondition Failed if the key already exists. In your own design, that is a compare-and-set on the metadata record: commit only if the record is still what you expected.

Failed uploads leave orphan chunks behind: chunks on disk that no metadata record points to. A garbage collector runs daily, finds chunks older than a day with no record, and frees them. Uploads still in progress register their upload id, so the collector skips their chunks.

Overwrites and deletes work the same way, so old chunks are freed later, never during the request. Reads run in reverse. The API server looks up the record, fetches chunks in parallel, checks each checksum, and streams the bytes back in order.

For an erasure-coded chunk, a reader can ask for 11 pieces and use the first 9 to arrive. That hides up to two slow servers, at the cost of two extra reads.

A blob store finds damage by checking a stored checksum on every read and in a slow background scan, then rebuilds bad pieces from healthy ones through a repair queue. Disks fail loudly, but they also fail quietly.

Loud failures are easy to see. A server stops sending heartbeats, small regular "I am alive" messages, and the placement service stops sending it new chunks. Deciding when to start rebuilding its data is the harder call.

Repair too early and you copy terabytes for a server that was only rebooting. Google's 2010 study of its storage cells says GFS, the Google File System, typically waits 15 minutes before recovering data from an unavailable node. In that study, less than 10% of unavailability events lasted longer.

Quiet failures are harder. A disk can return wrong bytes without any error, which is often called bit rot. Only a checksum catches it. AWS says S3 quickly detects and repairs lost redundancy, and regularly checks data with checksums.

The team builds the same two loops: a scrubber that reads everything slowly, and a repair queue that acts on what it finds. Here is the scrubber for one data node, with a stand-in node so it runs on its own.

import hashlib
import queue

class Node:  # stand-in for one data node
    def __init__(self, node_id, chunks):
        self.id = node_id
        self.chunks = chunks  # chunk id -> bytes on disk
        self.saved = {c: hashlib.sha256(b).hexdigest() for c, b in chunks.items()}

    def list_chunks(self):
        return self.saved.items()  # checksums saved at write time

    def read_chunk(self, chunk_id):
        return self.chunks.get(chunk_id)

def scrub_node(node, repair_queue):
    damaged = 0
    for chunk_id, expected in node.list_chunks():
        data = node.read_chunk(chunk_id)
        if data is None or hashlib.sha256(data).hexdigest() != expected:
            repair_queue.put({"chunk": chunk_id, "bad_node": node.id})
            damaged += 1
    return damaged

node = Node("node-07", {"a": b"invoice", "b": b"photo", "c": b"backup"})
node.chunks["b"] = b"phot0"  # one changed byte: silent bit rot
del node.chunks["c"]         # one chunk missing from disk
jobs = queue.Queue()
print("damaged:", scrub_node(node, jobs))
while not jobs.empty():
    print(jobs.get())

Run it and you should see damaged: 2, then one repair job for chunk b and one for chunk c, both from node-07. The changed byte and the missing chunk end up in the same queue.

What just happened? The if line is the whole idea. A missing chunk and a chunk whose fingerprint changed are treated the same way. The scrubber only reads and queues, so a bug in it cannot overwrite good data.

Repair workers take jobs from the queue. For a replicated chunk, they copy a healthy copy to a new node. For an erasure-coded chunk, they read 9 healthy pieces, recompute the lost one, and write it elsewhere. Then they update the metadata record.

• Repair the most damaged first. A chunk missing 5 pieces has 1 spare left, so it jumps ahead of one missing 1.
• Limit repair traffic, so rebuilding a dead server does not slow every customer's reads. f4's rebuilders throttle themselves for this reason.
• Never place a rebuilt piece in a zone that already holds 5 pieces of that chunk.
• Track time to repair as a key number, and alert when repairs fall behind new failures.

Beyond dead disks, the common problems are failure bursts, hot prefixes, accidental deletes, huge listings and bad deploys. Each needs a guard in the design, not a fix after the incident.

• Failure bursts. Google's 2010 study found most large bursts of failures tied to rack or multi-rack events. Guard: never put two pieces of one chunk on the same rack.
• Hot prefixes. One customer hammers one key range. Guard: automatic shard splits and per-bucket rate limits.
• Accidental deletes. A customer's script wipes a bucket. Guard: optional versioning, so a delete adds a marker and older versions stay readable.
• Huge listings. Listing a million keys in one reply times out. Guard: paged results with a continuation token, a marker that says where the next page starts.
• Bad deploys. New code writes wrong metadata. Guard: staged rollouts, metadata snapshots, and tools that compare records against chunks.

The team turns these guards into customer promises. It will keep each blob across three zones, check every byte with checksums, and offer versioning so a mistaken delete can be undone. It avoids quoting a number of nines until its repair times are measured.

Explain it as two systems, a metadata service and a data layer, joined by an upload path that commits metadata last. Then spend most of your time on durability, because that is what the interviewer is testing.

1. 1

   Pin down the requirements

   Ask about object sizes, read and write rates, total storage, durability target and whether customers need LIST by prefix and versioning.

2. 2

   Do rough numbers

   Estimate metadata size from object count, and raw disk from stored bytes times the overhead of your protection scheme.

3. 3

   Draw the parts

   Load balancer, stateless API servers, auth, placement, a sharded metadata service and data nodes spread over three zones.

4. 4

   Walk the write and read paths

   Chunk the file, write and verify chunks, commit metadata last, and explain how reads check every checksum.

5. 5

   Compare replication and erasure coding

   Give the overhead and failure tolerance of each, count pieces per zone, and say which data uses which.

6. 6

   Close with repair and hot spots

   Cover heartbeats, the repair delay, scrubbing, the repair queue, shard splits for hot prefixes and garbage collection of orphan chunks.