Monorepo Version Control at Scale Explained: Why Git Struggles with Agent Merges

Checkouts slow down, pushes get rejected and the servers that share the clone load fall behind. Monorepo version control at scale is the problem of keeping one shared repository fast under that load.

A monorepo is one repository that holds many teams' code. A merge is one change joining main, the shared branch everyone builds from. Coding agents are programs that write and submit code changes on their own.

Picture a common setup. A payments company keeps all its code in one git monorepo with about 40 GB of history. The setup is made up, but the problem is real.

In March the company gave its 180 engineers coding agents, and merges jumped from about 300 a day to about 4,000. CI, the system that builds and tests every change, now clones the repo about 2 lakh (200,000) times a day.

The platform team sees three problems. Agents keep hitting rejected pushes. CI machines spend minutes cloning. Eight git mirrors, read-only copies that share the clone load, cost about Rs 22 lakh a month.

Six words carry the rest of this guide. Each one is used the same way from here on.

• Commit: a saved snapshot of the code, with a pointer to the commit before it.
• Hash: a short fingerprint calculated from data. Change one byte and the fingerprint changes.
• Blob: git's word for the contents of one file version.
• Tree: a folder listing that maps each name to the hash of a blob or another tree.
• Ref: a named pointer to a commit. The branch main is a ref.
• Clone: a copy of a repository made on your machine.

It is version control built for one enormous repository with a very high change rate. The main ideas are lazy fetching, sparse checkouts, server-side commits and layered caches.

Lazy fetching means downloading an object only when something needs it. A sparse checkout puts only chosen folders on disk. Server-side commits mean a central service writes new commits itself. Layered caches keep copies of objects close to the machines that ask for them.

Think of a large public library. Plain git hands every reader a photocopy of every book, plus every earlier edition. A lazy system lets readers borrow only the pages they need, from a branch library near home.

The mapping: the books are blobs and trees, the photocopy is a full clone, the borrowed pages are a sparse, lazy checkout, and the branch library is a regional cache. Where the picture breaks: a library lends one copy, while any number of machines can cache the same object at once.

Git assumes each clone needs a complete, standalone copy, and that work is shared by pushing. Both assumptions get expensive when clients are short-lived jobs that are always online and merge constantly.

Every clone carries the whole history

A normal git clone downloads every commit and every version of every file. That is what lets you work on a plane. A CI job that lives for ten minutes gains little from it.

The 13 October post lists git's tools for working on a slice: shallow clones, sparse checkouts and promisor remotes. A promisor remote is a server that promises to send missing objects later, which is how partial clone works. The post says the team used all of them and still ran several hundred internal git mirrors at its peak.

Every push races every other push

By default, git accepts a push to main only if your commit is built on main's current tip. If someone else pushed first, the server rejects yours. You must fetch, rebase and push again.

To rebase is to replay your change on top of the newest main. In our test on git 2.55.0, the second of two pushes was rejected with the message "fetch first". It went through after a fetch, rebase and retry.

Sapling's scale docs describe this race in git and Mercurial, another version control tool. At 300 merges a day, the payments team barely noticed it. At 4,000, one merge lands every 22 seconds on average, and agents lose the race again and again.

It moved almost all version control work to its servers. The 13 October post says every commit, rebase and merge happens on the server, so there is nothing to push.

The post reports these numbers for its own monorepo. Merges grew from about 2,500 a day in May 2026 to over 30,000 a day by September. That is one merge about every three seconds.

The main branch history is almost 100 GB, and a compressed working copy is over 25 GB. CI alone runs about 25 million checkouts a day. The post says fetching that naively would mean over 600 PB of git traffic a day.

You can check that last number yourself: 25 million checkouts times 25 GB is 625 million GB, which is 625 PB (petabytes, millions of gigabytes).

After the switch, the post reports an infrastructure footprint 200 times smaller than its old git mirrors, in CPU and memory. CI and research checkouts take seconds rather than minutes. These are the company's own figures, not independent benchmarks.

The post says the system is by no means a replacement for git in general use. It expects this class of problems to become more common over the next six to 12 months. That is a forecast, not a measured trend.

Partial clone limits which objects git downloads. Sparse checkout limits which files appear in your working folder. You usually want both, because a sparse checkout on a full clone still downloads everything.

Git's sparse-checkout command chooses folders in its default cone mode, where naming a folder includes everything under it. A partial clone with the filter blob:none downloads commits and trees but skips blobs until a command needs them. A shallow clone instead keeps only the newest commits.

Pause and predict: on a full clone, how many file versions does a sparse checkout of two folders download? We ran it on a test monorepo with six folders, 182 files and 1,080 file versions in its history.

The answer to the prediction: all 1,080. Sparse checkout alone saved disk space in the working folder, not download time. Only the partial clone cut what crossed the network.

To try it on your own repository, replace REPO_URL and the folder names. The comments show what we saw on the test repo.

git clone --filter=blob:none --sparse REPO_URL demo
cd demo
git sparse-checkout set payments/upi libs/ledger
git sparse-checkout list
# libs/ledger
# payments/upi
git cat-file --batch-all-objects --batch-check='%(objecttype)' | grep -c blob
# 62   (blobs on disk; a full clone had 1080. Git may first warn about promisor remotes.)
du -sk .git/objects/pack
# 320  .git/objects/pack   (size in KiB, varies a little; a full clone had about 900)

Two details matter in practice. A normal checkout fetched all 182 missing blobs in one batch, but git log -p over 8 commits started 8 separate fetches. Git's docs warn that fetching objects one at a time tends to be slow.

The server must also allow filtering. Against a git server without uploadpack.allowFilter set, the clone printed "filtering not recognized by server, ignoring" and downloaded all 1,080 blobs.

The client downloads almost nothing up front and asks for each tree and blob only when needed. Every object is named by the hash of its contents, so the client can tell what it already has.

The post describes its client starting from a root tree and requesting missing trees recursively. It adds that this is resumable, because there is no single multi-gigabyte download. Here is the idea as a runnable Python model.

import hashlib, json

server = {}  # the cache tier: object hash -> bytes (stands in for the network)
def put(obj):
    data = obj if isinstance(obj, bytes) else json.dumps(obj, sort_keys=True).encode()
    h = hashlib.sha256(data).hexdigest()[:8]
    server[h] = data
    return h

# A tiny repo: a tree maps each name to ["tree" or "file", hash]
upi = put({"pay.py": ["file", put(b"print('pay')\n")]})
cards = put({"card.py": ["file", put(b"print('card')\n")]})
ledger = put({"book.py": ["file", put(b"print('book')\n")]})
root = put({"payments": ["tree", put({"upi": ["tree", upi], "cards": ["tree", cards]})],
            "libs": ["tree", put({"ledger": ["tree", ledger]})]})

local_store, fetched = {}, []
def get(h):
    if h not in local_store:              # miss: ask the cache tier once
        fetched.append(h)
        local_store[h] = server[h]
    return local_store[h]

def checkout(tree_hash, wanted, path=""):
    tree = json.loads(get(tree_hash))
    for name, (kind, child) in tree.items():
        full = f"{path}/{name}".lstrip("/")
        if not any(full == d or full.startswith(d + "/") or d.startswith(full + "/")
                   for d in wanted):
            continue                      # outside the sparse set: never fetched
        if kind == "tree":
            checkout(child, wanted, full)
        else:
            print("write", full, "->", get(child).decode().strip())

checkout(root, ["payments/upi", "libs/ledger"])
print(len(fetched), "of", len(server), "objects fetched")

It prints the two wanted files, then "7 of 9 objects fetched". What just happened? The continue line skipped payments/cards, so its tree and file were never requested. The get function fetches each hash at most once per machine.

Because a hash names exact contents, a cached object can never go stale. That is why any machine or cache tier can keep copies without asking whether they are current.

The server rebases your change onto the newest main for you, instead of rejecting it. In Sapling's design, the server moves your change to the top of main, and the push fails when someone else edited a file you also touched.

Sapling's docs describe this server-side rebase. They add that Sapling can call code generators during a rebase to resolve conflicts in generated files. The 13 October post says commits, rebases and merges run on the server, but does not spell out its conflict rule.

Compare-and-swap moves a ref only if it still points where you expected. If it moved, the update fails, and you read the new value and retry. No lock is held while you work.

The post says moving a ref is a compare-and-swap on a single Spanner row, separate from creating changesets. Spanner is Google's distributed database. A changeset is the post's unit of change: like a commit, but stored as the files changed since its parent.

The post says creating changesets needs no locking or cross-server coordination. It says the team could not effectively reproduce the usual global lock on the merge queue. So concurrent changes can conflict, and it relies on post-merge and pre-deploy testing.

import threading

class RefRow:                     # one database row holding the main ref
    def __init__(self, commit):
        self.commit, self._lock = commit, threading.Lock()
    def read(self):
        return self.commit
    def compare_and_swap(self, expected, new):
        with self._lock:          # the database makes this one step atomic
            if self.commit != expected:
                return False
            self.commit = new
            return True

history = {"a71b": set()}         # commit -> files it changed vs its parent
parent = {"a71b": None}
def files_changed(since, head):   # files changed after `since`, up to `head`
    out = set()
    while head != since:
        out |= history[head]; head = parent[head]
    return out

def land(base, files, name, main, before_swap=None):
    while True:
        head = main.read()
        if files_changed(base, head) & files:
            return f"{name}: conflict, send back to the author"
        new = f"{name}@{head}"    # server-side rebase: a new commit on head
        history[new], parent[new] = files, head
        if before_swap:
            before_swap(); before_swap = None   # a rival lands first, once
        if main.compare_and_swap(head, new):
            return f"{name}: landed as {new}"
        print(f"{name}: swap lost, main moved to {main.read()}; retrying")

main = RefRow("a71b")
rival = lambda: print(land("a71b", {"libs/ledger/book.py"}, "rival", main))
print(land("a71b", {"payments/upi/pay.py"}, "agent1", main, before_swap=rival))
print(land("a71b", {"payments/upi/pay.py"}, "agent2", main))

It prints four lines. A rival lands first and agent1 loses the swap, then agent1 retries and lands on top of the rival. Then agent2, which edited the same file as agent1, gets a conflict.

Plain git already checks the old value before moving a ref, under a short-lived lock file. On git 2.55.0, update-ref with a stale expected commit failed with "is at" one commit "but expected" another. A push with --force-with-lease against a stale value was rejected as "stale info".

So the real difference is not the check on main. In git, the loser repeats a network round trip of fetch, rebase and push. On the server, the retry is a cheap server-side step.

Requests go to a cache on the same machine first, then a regional cache, and rarely to the main server. Objects never change once written, so any tier can keep copies safely.

The post describes three tiers. A daemon, a background program, runs on each machine and starts from a periodically refreshed snapshot. Regional caches stream objects from memory, and authoritative servers read from Spanner or cloud storage on a miss.

The post says each tier keeps a cache hit rate of over 99 percent and needs no long-lived state. It says the tiers are strictly enforced to isolate failures, so that a failing regional cache does not overwhelm the servers above it.

The post also computes blame, path history and commit-graph skip lists as derived data. Derived data means indexes the server computes from each commit's parents, on demand, and stores for reuse.

Scalar, a tool that ships with git, turns on most of them with one command. We ran scalar clone with git 2.55.0 and read the settings it wrote.

After scalar clone, the working folder held only the two top-level files, and only 2 blobs were on disk. You widen it with git sparse-checkout set, exactly as before.

Git 2.55.0's release notes say the fsmonitor daemon is now implemented for Linux, so scalar enabled it there. After the first run, git status got the list of changed files from the daemon instead of scanning the whole folder.

One detail is undocumented as of 19 October 2026, so pin your version. Scalar's docs page does not list core.fsmonitor, but scalar's source sets it to true wherever the daemon is supported.

Git's docs warn that on Linux the daemon uses inotify, which has a per-user watch limit, "typically 8192". Our test machine showed 64809. Check yours with cat /proc/sys/fs/inotify/max_user_watches before turning it on for a big repo.

A commit-graph is a file that caches the shape of history for fast traversal. Scalar's scheduled maintenance keeps it fresh, and git commit-graph verify checks it.

They tackled the same problems years earlier, and the 2026 post names both as prior art. Sapling's docs say clones do not download all file and tree data; the client fetches it when needed.

Google's Piper

A 2016 Communications of the ACM paper by Rachel Potvin and Josh Levenberg describes Piper, Google's in-house version control system. Its abstract says the repository serves tens of thousands of developers around the world.

Meta's Sapling and the prior-art credit

Sapling's scale docs cover lazy history, on-demand file and tree fetching, sparse checkouts and server-side rebase. The 2026 post's footnote names its server, Mononoke.

On 14 October 2026, an engineer who helped build Mononoke wrote that the original post did not mention Mononoke. On 15 October the post was updated. It now says Mononoke pioneered many of its techniques, in particular its Bonsai data model.

Almost certainly not yet. The 2026 system is in-house, and its authors say it is no general replacement for git. Partial clone, sparse checkout and scalar are the cheaper first steps.

For the payments team, the plan in order of effort looks like this.

1. 1

   Measure first

   Record clone time, rejected pushes and mirror cost for one week, so you can tell later whether each step helped.

2. 2

   Turn on partial clone in CI

   Clone with --filter=blob:none, and confirm the server allows filtering by checking that no "filtering not recognized" warning appears.

3. 3

   Add cone-mode sparse checkouts

   Give each CI job only the folders it builds, using git sparse-checkout set.

4. 4

   Use scalar on developer machines

   Scalar clone sets partial clone, sparse checkout, scheduled maintenance and, where supported, fsmonitor in one step.

5. 5

   Consider server-side landing last

   Only if rejected pushes still climb, look at a merge queue or server-side rebase, and keep tests running after merge.

Frame it as two problems, reading the repository and writing to it, then add caching.

1. 1

   State the numbers

   Give repository size, merges per day and checkouts per day. 30,000 merges a day is one every three seconds.

2. 2

   Explain why full clones fail

   Every job downloads history it never reads, and mirrors drift behind main.

3. 3

   Make reads lazy and sparse

   Name objects by content hash. Fetch the root tree, then only the needed folders.

4. 4

   Move writes to the server

   The server rebases each change onto main and rejects changes that touch the same files.

5. 5

   Guard main with compare-and-swap

   Move the main ref only if it still points at the expected commit. Otherwise retry.

6. 6

   Add caching tiers and trade-offs

   Use machine, regional and central tiers. Admit that clients must be online and tests still run after merge.