Design Dropbox or Google Drive: File Sync System Design

Small edits become slow uploads, storage fills with copies, and two people's saves can overwrite each other. A simple app knows only that a file changed, not which part, so it sends every byte again.

Picture a common setup, the kind file sync system design is for. A team builds a sync service for small offices, and each office keeps one shared folder the same on every laptop. The setup is made up, but the problems are real.

The team's first desktop app was simple: when a file changed, upload all of it. Then one office, a textile exporter in Chennai, started using it for its price list: a 300 MB CSV file, plain text with one row per product.

• Slow uploads. A staff member fixes one price, and the app sends all 300 MB again. On a 10 Mbps (megabits per second) office line, that takes about four minutes.
• Wasted storage. Fifty staff keep the same 80 MB product catalogue, and the service stores fifty full copies.
• Lost work. Two staff edit the same quotation, one of them offline at a client site. The later save quietly replaces the earlier one.

The last problem is the one that loses customers. This guide fixes all three, one design choice at a time, on that same office folder.

File sync system design is the plan for keeping one folder the same across many devices and a server. It covers how files are split, stored, announced to other devices and merged.

Sync means making copies match. Every device keeps a local copy on its own disk. The server holds the true record of what each file should contain right now.

Before drawing any boxes, the team writes down what the service must do.

• Upload, download and edit files from a desktop app, a phone or a browser.
• Show a change from one device on every other online device within a few seconds.
• Keep working offline, and sync the edits once the network is back.
• Keep old versions for 30 days, so a bad save can be undone.
• Never lose an edit, even when two people change one file at the same time.

Two goals shape everything else. Uploads should send only what changed. And the same bytes should never be stored twice inside one customer's account.

Splitting files into chunks lets a service upload only the parts that changed, and store each unique part once. A chunk, also called a block, is one small piece of a file.

A bookbinder sews a book in signatures: small bundles of folded sheets, stitched together one after another. When water spoils one bundle, the binder unpicks that signature and sews in a fresh one. Nobody reprints the whole book.

Map it part by part. The book is the file, and each signature is a chunk. The binder's ordered list of signatures is the file's ordered list of chunk names.

Where the picture breaks: a binder can see which bundle is damaged. A sync app cannot see that, so it re-reads the file and compares the chunk names.

Each chunk is named by its hash. A hash function, such as SHA-256, turns any bytes into a short fingerprint. The same bytes always give the same fingerprint, and different bytes practically never do.

The ordered list of chunk hashes is called a blocklist. A file version is simply its blocklist. Dropbox's July 2014 engineering post describes this design: 4 MB blocks, each hashed with SHA-256, the last block possibly smaller.

The team's client builds the blocklist like this. The first function reads a file 4 MiB at a time and hashes each piece. The second keeps only the hashes the server does not already have.

import hashlib

BLOCK_SIZE = 4 * 1024 * 1024          # 4 MiB per chunk

def blocklist(path):
    hashes = []
    with open(path, "rb") as f:
        while True:
            block = f.read(BLOCK_SIZE)
            if not block:                  # end of file
                break
            hashes.append(hashlib.sha256(block).hexdigest())
    return hashes

def blocks_to_upload(local_hashes, server_has):
    return [h for h in local_hashes if h not in server_has]

Two lines matter most. The call f.read(BLOCK_SIZE) returns at most 4 MiB, and an empty result means the file has ended. The last line is the whole saving: any chunk the server already has is skipped.

A note on units. 4 * 1024 * 1024 is 4 MiB (mebibytes), or 4,194,304 bytes, a little more than 4 million bytes. Dropbox's post just says 4MB.

Now a price fix that keeps the same number of characters costs one 4 MiB chunk instead of 300 MB. On the same 10 Mbps line, the upload drops from 240 seconds to about 3.4 seconds, before any protocol overhead.

That saving assumes the edit changes bytes inside one chunk, as a plain-text CSV does. Many office formats, such as .xlsx workbooks, are compressed ZIP archives, so one small edit can change most of the file. A longer price is a problem too: it shifts every later byte, and the engineer section shows why that matters.

We checked: with openpyxl 3.1.5, editing one cell near the top of an 8.8 MiB workbook changed all 3 of its 4 MiB chunks. Measure with your customers' real files before promising savings.

Chunk names also give you deduplication for free. Deduplication means storing identical data only once. When fifty staff save the same catalogue, the server sees fifty identical blocklists and keeps one set of chunks.

A Dropbox-style upload sends the blocklist first and the bytes second. The server says which chunks it lacks, the client uploads only those, and then a commit saves the new version as the latest one.

The team splits the server into two services. The metadata service knows files, folders, versions and blocklists. The block service only stores and returns chunks by hash, using object storage, a cheap store built for large blobs of bytes.

Follow the arrows below for one save of the price list. Notice that the client asks to commit twice: once to learn what is missing, and once to finish.

Dropbox's 2014 post describes the same order. The client first tries to commit the blocklist, the server replies need blocks, and the client sends them to the block server. Then it commits again, and that commit should succeed.

The commit comes last for a reason. Until every chunk is safely stored, the new version must not exist. Otherwise another laptop could see a file whose bytes are not there yet.

Chunks also make uploads resumable. If the laptop's network drops after chunk 40 of 72, the next attempt asks again and sends only the 32 chunks still missing.

Other devices learn about changes by reading an ordered change journal from a saved cursor. A notification tells them when to read, but the journal tells them what changed.

Every commit adds one row to the journal. The row holds a namespace (one shared folder tree), a path, the new blocklist and a number that only goes up. A cursor is the last journal number a device has applied.

Dropbox's 2014 post calls this the server file journal: append-only, one row per file version, numbered in increasing order within each namespace. Each desktop client keeps one cursor per namespace.

Each laptop keeps its cursor on disk. When it wakes up, it asks for every row after the cursor, applies them in order and saves the new cursor. A laptop switched off for a week catches up the same way.

Google Drive's public API works on the same idea, as of 19 October 2026. A client gets a start page token, lists the changes after it, and stores the new start page token for next time.

Its push notifications carry no change details. They only say that new changes are available, so the client then fetches them from the change list.

The team uses the same split. Idle laptops hold a long poll, a request the server keeps open until it has news, to a notifier service. Dropbox's 2014 post describes the same longpoll connection.

The notifier sends a tiny wake-up message, never file data, and the laptop pulls the details from the journal. Look at the metadata service in the diagram: every commit passes through it, and every device reads the journal from it.

A sync client compares three views of each file: what is on disk, what is on the server, and what both looked like at the last successful sync. The third view tells it who changed.

The team's first client compared only disk and server. When a file was on the server but not on disk, it could not tell two cases apart. Either someone added it remotely, or the user deleted it locally.

Dropbox's rewritten sync engine, Nucleus, solves this with three trees, per its April 2020 testing post. The remote tree is the latest cloud state, and the local tree is the last state seen on disk.

The synced tree is the last known fully synced state between them. The post says each of its entries works as a merge base, the shared starting point that version control tools like Git compare against.

Here is the core decision for one file. Each of the last three arguments maps a path to that file's blocklist hash. A missing path gives None, meaning the file does not exist there.

def decide(path, local, remote, synced):
    disk, server, base = local.get(path), remote.get(path), synced.get(path)
    if disk == server:
        return "in sync"
    if disk == base:
        return "apply server change"   # only the server moved
    if server == base:
        return "send local change"     # only this laptop moved
    return "conflict"                  # both moved since last sync

The order of the checks is the point. If the disk still matches the last sync, only the server moved, so the client applies the server's change, even a deletion. Only when both moved is it a conflict.

What just happened? Run it on the confusing case: no file on disk, file on server, nothing in the synced view. It returns apply server change, a remote add. Put the file in the synced view, and it returns send local change: the user deleted it.

The same logic as a chart is below. Start at the top and answer one question at a time for each file.

A file sync service should keep both versions when two devices changed the same file since the last sync. It saves one edit as a separate conflicted copy and never silently throws work away.

A conflict means two edits started from the same version. On the server, the team catches it with a parent version. Every commit says which version it edited, and the server accepts it only if that is still the latest.

This is called optimistic concurrency control. Nobody locks the file while editing, and the rare clash is caught at commit time. The later commit gets a conflict reply instead of overwriting the earlier one.

Dropbox's help centre page, updated 15 May 2025, describes what users then see. The editor's username, the words conflicted copy and the save date are added to the file name. The last version saved becomes the conflicted copy.

The page lists three causes: two people editing at once, one person editing offline, and an app autosaving a file left open on another computer. It advises comparing both versions and merging them by hand.

The team now does the same for the quotation. The offline edit becomes a second file beside the first, and both staff get a notice to compare them.

Automatic merging works only when the system understands the file. A sync service sees spreadsheets, photos and design files as plain bytes, so it cannot safely combine two edits. Live shared editors merge inside the editor instead.

That completes the core design. Beginners can stop here. The rest is for engineers: where fixed chunks fail, how downloads start early, clashes that are not about contents, deduplication risks and storage maths.

Fixed-size chunks fail because an insert shifts every later byte, so every later chunk holds different bytes and gets a new hash. Content-defined chunking fixes this by cutting where the content says, not at fixed offsets.

Pause and predict. A 2 MiB file is cut into fixed 8 KiB chunks, 256 in all. You insert ten bytes at the very start. How many chunks does the server not already have?

How content-defined chunking picks its cuts

Content-defined chunking uses a rolling hash. A rolling hash is a fingerprint of the last few bytes that is cheap to update as you move forward one byte. When the fingerprint matches a chosen pattern, the chunker cuts there.

Because each cut depends only on nearby bytes, the cuts after an insert land in the same places in the content as before. Those chunks keep their old names, so the server already has them.

The code below uses a Gear-style rolling hash. Each byte shifts the 64-bit fingerprint left by one bit and adds a random number for that byte value. After 64 steps, a byte has shifted out completely.

import hashlib, random

random.seed(7)
GEAR = [random.getrandbits(64) for _ in range(256)]   # one random number per byte value
MIN, MAX = 2 * 1024, 64 * 1024                       # smallest and largest chunk

def cdc_chunks(data):
    chunks, start, h = [], 0, 0
    for i, b in enumerate(data):
        h = ((h << 1) + GEAR[b]) & 0xFFFFFFFFFFFFFFFF  # old bytes fall off after 64 steps
        size = i + 1 - start
        if (size >= MIN and h >> 51 == 0) or size >= MAX:  # top 13 bits zero: cut
            chunks.append(data[start:i + 1])
            start, h = i + 1, 0
    if start < len(data):
        chunks.append(data[start:])
    return chunks

def fixed_chunks(data, size=8 * 1024):
    return [data[i:i + size] for i in range(0, len(data), size)]

def names(chunks):
    return [hashlib.sha256(c).hexdigest() for c in chunks]

def new_chunks(before, after):
    old = set(names(before))
    return sum(1 for h in names(after) if h not in old)

file = random.randbytes(2 * 1024 * 1024)               # a 2 MiB file
edited = b"ten bytes!" + file                          # insert 10 bytes at the start

for label, split in [("fixed 8 KiB", fixed_chunks), ("content-defined", cdc_chunks)]:
    a, b = split(file), split(edited)
    print(f"{label}: {len(b)} chunks, {new_chunks(a, b)} new after the insert")

Running it on Python 3.13 prints two lines. Fixed 8 KiB: 257 chunks, 257 new after the insert. Content-defined: 202 chunks, 1 new after the insert.

What just happened? Every fixed chunk moved by ten bytes, so all 257 names changed. The content-defined chunker re-cut only the first chunk, then found its old cut points again.

One run proves little, so we repeated it at 200 random places in a 512 KiB file, inserting 1, 10 or 100 bytes. The content-defined chunker produced one new chunk 197 times and two new chunks 3 times.

So the honest claim is one or two new chunks around the edit, not exactly one.

Where the classic numbers come from

The 2001 LBFS paper, a low-bandwidth network file system, set the classic recipe. It ran Rabin fingerprints, an older rolling hash, over every 48-byte window of a file.

LBFS cut wherever the low 13 bits of the fingerprint equalled a chosen value, which the paper says gives an expected chunk size of 8 KB. It set a minimum of 2K and a maximum of 64K, the same limits our code uses.

Faster chunkers followed. The FastCDC paper (USENIX ATC 2016) reports about 10 times the speed of the best open-source Rabin-based chunking, and about 3 times Gear-based chunking, with nearly the same deduplication.

Note what Dropbox's 2014 post does not say. It describes fixed 4 MB blocks, not content-defined ones. Treat that as the 2014 design; Dropbox has rewritten its sync engine since.

How rsync handles inserts with fixed blocks

rsync, a common file copying tool, takes a different route. It keeps fixed blocks on the receiving side but searches the sender's file for matches at every byte offset, not just at block edges.

Its technical report pairs a weak 32-bit rolling checksum, a cheap fingerprint, with a strong 128-bit MD4 checksum. The weak one finds candidates fast, the strong one confirms them, and the whole exchange needs one round trip.

We checked this on rsync 3.2.7, the Ubuntu 24.04 package. We inserted ten bytes at the start of a 20,000,000-byte file and synced it over an old copy with --no-whole-file --stats.

rsync reported 10 bytes of literal data and 20,000,000 bytes of matched data, and the copies were identical. The flag matters, because rsync sends whole files by default when both paths are local.

Strong checksums have moved on since the report. rsync 3.2.7 lists xxh128, xxh3, xxh64, md5, md4 and sha1 in its --version output, and its manual says to check that list on your own build.

So why not just use rsync? It needs the old copy on the other side to compare against. A sync server with deduplication instead needs chunk names that stay stable, so any device can ask whether a chunk exists.

They can prefetch chunks that are already stored while the uploader is still sending the rest. Dropbox's 2014 post calls this streaming sync, and it keeps the commit-last rule intact.

In Dropbox's design, a failed commit, the one that returned need blocks, is remembered in a cache (memcache) by the metadata server. When another client lists changes, it gets those not-yet-committed blocklists too.

That client prefetches the listed chunks into a cache. The file still becomes a journal row only at the commit, but by then the downloader is nearly done.

Dropbox's post gives up to 2x faster sync across clients as the best case. Its own test measured roughly 25%: 293 seconds instead of 383 for a 500 MB file.

For the office, that means a phone can start pulling the new price list while the laptop is still uploading. Those are Dropbox's 2014 measurements; measure your own link before you promise numbers.

Deletes, renames, folder moves and names that differ only in case all clash without anyone editing bytes. A client needs a fixed rule for each, decided before the first customer hits it.

• Edit versus delete. The decide function above returns conflict when the disk has an edit and the server has a deletion. The team's rule keeps the edit, because losing data is worse than an unwanted file.
• Two renames. Both sides rename the same file differently. Pick one name by a fixed rule, and log the choice.
• Folder moves. One side moves a folder while the other adds a file inside it. Track files by stable IDs, not paths, so the new file follows the folder.

Dropbox's 2020 post explains why IDs matter. Its older engine keyed files by full path, so a rename was stored as a delete plus an add. In Nucleus, each file has a unique ID, and a move just updates its attributes.

Names that differ only in case

By default, Windows treats file names as case-insensitive, so FOO.txt and foo.txt are the same file, per Microsoft's WSL documentation. Linux treats them as two different files.

Windows can mark single folders as case-sensitive, but that is off by default, so plan for the default.

We checked the Linux side on an ext4 disk, a common Linux file system. Creating Invoice.xlsx and then invoice.xlsx left two files in the folder.

Now sync that folder to a Windows laptop. Both names map to one file there, so the second download would overwrite the first. Catch the clash before downloading and rename one copy instead.

Deduplication across all customers leaks information and can hand out files. Keeping it inside one account, here one office, closes both holes at the cost of some extra storage.

The first hole is a side channel, information leaked by how a system behaves. If the app skips uploading any chunk some customer already stored, the skip itself is visible. Another customer can upload a guessed file, watch whether their own app sends the bytes, and learn whether someone else already stored it.

A 2017 arXiv paper by Zuo and others says the occurrence of deduplication can be easily identified by monitoring network traffic. It cites earlier tests by Harnik and others on popular cloud storage services.

The second hole is ownership. A 2011 paper by Halevi, Harnik, Pinkas and Shulman-Peleg shows that someone who knows only a file's hash can convince such a server they own the file, then download it.

Their fix is a proof of ownership: the client must show it holds the actual data, not just a short fingerprint. The team chose the simpler fix and deduplicates only within one office's account.

You scale it by keeping metadata and chunk bytes apart. Metadata is small and must be strictly ordered, while chunks are huge, never change once written, and fit cheap object storage.

The team does a rough estimate. Say 3,000 offices with 40 staff each, so 1.2 lakh (120,000) users. If each user keeps 10 GB, that is 1.2 PB (petabytes, a million GB each) before deduplication.

Metadata is far smaller. With 4 MB chunks, 1.2 PB is at least 300 million chunks, and more when files are small, since each small file is a chunk of its own. Assume about 100 bytes per chunk record, a made-up round number: a 32-byte SHA-256 hash plus IDs and sizes. Then the chunk index is at least about 30 GB.

Chunks are immutable, which means they never change. A chunk named by its hash cannot change, because new bytes would get a new name. That makes chunks easy to cache and copy to another region.

The team shards the metadata by namespace. A shard is one slice of the data on its own database machine. One office's folder tree lives on one shard, so its commits stay on one machine and in strict order.

• File and folder records, versions and blocklists: a relational database (rows in tables, queried with SQL), sharded by namespace.
• The change journal: rows in that same database, numbered in order within each namespace.
• Chunk bytes: object storage, named by hash, copied to a second region.
• Popular chunks, like a shared catalogue: a CDN, servers near users that keep copies, close to the offices.

Old chunks need garbage collection, a cleanup job that deletes chunks no kept version points to. The team's job waits a day after a chunk loses its last reference, because an upload in progress may be about to reuse it.

Start from one small edit to one big file, and let each design choice fix a problem you can name.

1. 1

   Clarify the scope

   Ask about file sizes, number of devices, offline use, sharing and how long to keep versions. Write the answers down where the interviewer can see them.

2. 2

   Chunk and hash

   Split files into chunks named by their hash. Explain blocklists and deduplication, then say why content-defined chunks survive inserts.

3. 3

   Split metadata from blocks

   Draw a metadata service with its database, and a block service on object storage. Say why the two scale differently.

4. 4

   Walk through an upload

   Blocklist first, missing chunks next, commit last. Point out that this also makes uploads resumable.

5. 5

   Walk through a download

   Describe the journal, the cursor and the wake-up messages. Say the journal is the source of truth and notifications are only hints.

6. 6

   Handle conflicts

   Three-way comparison on the client, parent versions on the server, and conflicted copies for people to resolve. Add rules for deletes, renames and case clashes.

7. 7

   Estimate and scale

   Do rough storage maths out loud, then cover sharding by namespace, garbage collection, per-account deduplication and a CDN for popular chunks.