Git Hosting Has a Scale Problem. Do You?

Git hosting is straining under AI agent traffic. Learn which limits affect your team and how file APIs, safer transcripts and reviews grounded in intent can help. The hosts examine storage changes at GitHub, GitLab and Cursor Origin, explain why Anthropic's Scion serves an exceptional workload, and weigh Entire's approach to capturing agent sessions. For most teams, token costs, review capacity and API budgets become constraints long before Git does.

Summary

Your agents can open more pull requests in a morning than your team can review in a week. Pushes still go through in a few seconds, and nobody on your team is complaining that Git is slow. Yet several major Git hosts are rebuilding their storage for agent traffic, and one AI lab has left Git altogether, so it’s easy to conclude that Git is broken. For most teams at ordinary workloads, it isn’t. The scale problem is real, but it falls on the companies that host Git for millions of users and their agents, and they are the ones fixing it.

GitHub reports pushes up 4.9 times year over year. Anthropic’s merges went from about 2,500 a day to over 30,000 in four months. The hosts are converging on one direction: separate durable storage, the compute that serves Git, and maintenance, so that adding read capacity stops slowing writes. For most engineering teams, other limits arrive first: tokens, review capacity, and knowing why an agent changed what it changed. The new idea worth watching is storing the agent’s session next to the commit, though storing transcripts still risks exposing secrets.

Git was built for offline work, and the hosts pay for it

The pressure starts with a design choice made in 2005.

Is Git itself the bottleneck for agent workloads?

For most teams at ordinary workloads, no. Git’s distributed design makes centralized hosting expensive at very large scale, and that cost falls mostly on the platforms.

Git was created in 2005 for distributed Linux kernel development, after the project lost free access to BitKeeper. It assumes you clone everything, keep a full copy, and can work offline. Every clone can act as a server, which is how forking and mirroring work, and it’s why Git became the foundation of open source. Centralized systems such as ClearCase and Perforce made the opposite bet: stay connected to one server. Perforce is still going strong in some big companies, because inside one organization you can usually stay connected.

Hosting a distributed protocol centrally has always been hard for GitHub. Its engineers have given conference talks about it for years, and GitHub rewrote libgit to make it more efficient. Agents just raised the scale. GitHub’s own numbers show pushes growing from 0.69 billion to 3.35 billion a month in a year, and 7.38 billion commits in September alone. An agent in a tight loop commits after nearly every action.

GitHub’s current storage design, Spokes, makes the trade-off clear. Each repository lives as several full copies on local disks, and a push counts only when a quorum of copies agrees. The copies provide both durability and read capacity, so in GitHub’s words: “adding replicas to absorb read load makes writes slower.” Picture a panel of notaries, each holding a full copy of the contract. Add notaries and more people can read at once. But every new page now waits on a bigger panel, and the slowest notary sets the pace.

That trade-off hurts in two places. A busy monorepo needs more copies than the protocol can coordinate cheaply, and millions of tiny throwaway agent repos each pay for copies that sit idle. Which one applies to you depends on your workload, not your headcount. At ordinary workloads, most teams run into neither.

Everyone moves the heavy work off the push

The fixes share a direction, even where the details differ.

What are GitHub, GitLab and Cursor actually changing?

They separate durable storage from the compute that serves Git requests, and they move maintenance off the serving path. The ref update still has to be coordinated before a push is acknowledged.

Git ships changes as packfiles. When you push, the receiving server checks the incoming pack. It either indexes the pack and keeps it, or, for a small push, unpacks it into loose objects. Git’s receive.unpackLimit sets that threshold. The server then checks that the new objects connect to history and updates the ref. Automatic garbage collection and repacking can follow a push once their thresholds are crossed. On a host that keeps several full copies, much of this work happens against every copy, and all of it is the receiver’s work, not yours.

Each vendor applies that idea differently:

• GitHub moves authoritative repository data into Azure Blob Storage, with caching compute workers in front and maintenance on separate workers. The ref update stays coordinated on the push’s critical path, while storing objects, connectivity checks and secret scanning can run in parallel with other writes. GitHub claims “up to 35 times higher write throughput” in internal benchmarks, and the rollout isn’t finished.
• Cursor Origin, per Vicent Martí’s storage design post, keeps a write-ahead log in S3-compatible storage and treats repos on disk as a warm cache. A push atomically updates the log’s index with a compare-and-swap on S3 and is persisted before it is acknowledged. Cursor measured up to 120 pushes a second on S3 Standard and over 300 on S3 Express One Zone, on its own monorepo.
• GitLab’s next-generation SCM separates compute from storage, and the two tiers talk over the S3 API. It’s in private beta.
• Entire describes regional content-addressed storage with compare-and-set ref updates.

As far as we can tell, Cursor got to the write-ahead log first and GitHub is catching up. Two caveats apply to every number above. They’re vendor benchmarks, and nobody has run an independent comparison. And they measure pushes, not merges. main is still one ref, so merges into it still land one after another.

If storage engines converge, Git speed stops being the reason to pick a host. The difference moves to the workflow on top: pull requests, CI, policy, and what the agent can ask the server. For Origin, we couldn’t find much beyond speed yet. Its own developer, asked on launch day how it differs from GitHub, answered: “Today, very little.”

Don’t clone, ask

For a platform team, the faster push isn’t the change that matters most. Not cloning at all is.

How can an agent read one file without cloning the whole repository?

Ask the server. GitLab is building APIs shaped for agents, and GitHub and GitLab already have file APIs you can use today.

An agent on your laptop usually has a copy of the repository. An agent that runs on its own, say in CI or on a schedule, often doesn’t. A naïvely configured job may clone the whole repository to check one file on one branch. Shallow and partial clones fetch less, but the job still sets up a local repository to answer one question. GitLab calls this the clone tax: “One agent invocation today can mean 5GB to 10GB transferred and 30+ seconds of setup, just to answer a single question.” Its answer is server-side queries: one batch read instead of a full clone, a diff-stat request instead of clone-then-diff, a last-commit-for-path lookup instead of clone-then-blame. The backend does the processing, and the agent gets only the slice it asked for.

GitLab’s own figures are up to 50 times faster tasks and up to 1,000 times less network traffic. GitLab adds the caveat itself: “These are ceilings, not guarantees.” If you’re on GitLab, the beta is worth joining.

You don’t need to wait for it, though. GitHub’s Git database API and GitLab’s repository files API already let an agent read a file or a tree without a clone. Datadog hit the same wall as a customer. Agents drove “an order-of-magnitude increase in Git traffic”, and cloning on demand from GitHub only moved the thundering herd upstream into rate limits. Datadog’s gitretriever puts a self-hosted mirror behind a small read-only HTTP API for the queries agents actually make.

The trade-off is that build and test steps still need a working tree. “Don’t clone, ask” helps audit and lookup jobs, such as checking how every repository in an organization is configured, or finding which repos still pin an old module version. Those jobs often consume the most of your API budget.

The agent’s session is the missing why

Entire is betting on something other than speed.

Should you store agent transcripts next to your commits?

The idea is sound. But raw transcripts can contain secrets, so plan for credentials and access before you turn it on.

Entire’s open-source CLI captures the agent session (prompts, transcript, files touched, tool calls) and links it to the commit. It started with a separate branch, entire/checkpoints/v1, and now uses a Git ref per checkpoint by default. Its founder’s line: “Git preserves what changed, but nothing about why.”

That “why” has real value. A thousand-line change lands with a pull request and a commit message, and neither captures the decisions behind it. Was the over-engineered part a first attempt, or did the engineer ask for it? The session shows how the change was made and what its owner asked for. A vibe-coding session full of “I don’t like that, change it” tells you little. A grilling session where someone made decisions and justified them is worth keeping.

The catch is everything else in the transcript. If the agent reads your API key, the key can end up in the session log. Entire redacts before it writes checkpoints, but its security docs say to “treat redaction as a safety net rather than a guarantee.” One hands-on reviewer found a full key in plaintext on a local snapshot branch even though .env was in .gitignore. Current docs exclude gitignored paths from those snapshots, but a key hardcoded in a tracked file still shows up there. A user who found secrets in pushed checkpoints wrote: “Locking this airtight is HARD.”

Two fixes don’t depend on Entire. First, get rid of static credentials. Static keys an agent can read can end up in its history, in .claude or .cursor JSON files and in other places you won’t think to check. Short-lived credentials don’t make a leak harmless: per AWS’s documentation, a temporary credential works until it expires or is revoked. They shrink the window to minutes or hours. Second, decide who can read transcripts. Read access to a repository isn’t the same trust boundary as the right to read raw agent history. Entire can push checkpoints to a separate repository with its own access.

Volume is the other open problem. A long chat can end in one tiny line of code. Milestone checkpoints or a summary of intent would be smaller, but someone pays for the tokens, and doing that on the push path slows the push. Git already has a little-known place to store such data, git notes. git-ai uses them for per-line attribution and keeps the transcript outside Git.

Anthropic left Git, and you can stay

Anthropic is in a league of its own, which is why its answer doesn’t transfer.

Do you need to replace Git like Anthropic did?

Unless your workload resembles Anthropic’s, with tens of thousands of merges a day into one monorepo, very likely not.

Anthropic’s post describes going from about 2,500 merges a day in May to over 30,000 in September, with several hundred Git mirrors running at the peak just to keep the system from melting. Its replacement, Scion, works a bit like Perforce crossed with streaming. Clients are always online and fetch only the files they read, and the server does commits, merges and rebases. Merges run in parallel without a global merge-queue lock, and Anthropic reports that Claude fixes semantic conflicts after the merge, for example a renamed function racing a new call to it. That works with a near-unlimited supply of tokens. Anthropic says its internal token spend “far exceeds what any other business would realistically want to spend”, and that Scion “is by no means a suitable replacement for git for general use.”

Google, Meta and Uber hit monorepo limits years ago without an unlimited token budget. The difference is how far along the AI adoption curve an organization is. AI today looks a lot like continuous integration around 2010, when CI was a new practice that some teams adopted and others didn’t. Some teams today are at the equivalent of nightly builds: pasting from ChatGPT now and then, unsure whether they’re allowed to share code at all. If you spend your day on X, YouTube and LinkedIn, it’s easy to think everyone works like Anthropic. Most organizations don’t. The future is here, but it isn’t evenly distributed.

AI adoption differs from CI in one way. CI spread step by step, because every step cost engineering time. With AI, a company can see “software factory” on YouTube and try to jump straight there, because the tokens do the work.

“But, Git still works for you as an end user, I think, for probably 99.9% of the people using it.”

So for now, this is the Git hosts’ problem. Watch where it goes. If it goes well, Git just gets faster for you, and you don’t have to change anything.

Tokens and review run out before Git does

If push speed isn’t your constraint, two other things probably are.

What actually limits how fast a team ships with agents?

Usually token cost first, then human judgment. Agents can push far more commits than anyone can review.

For most teams, token cost is the harder limit, and it’s easing as AI labs compete on price and local models get faster. Review capacity isn’t easing. Some builders who can now move ten times faster report that by the end of the day their brain has melted, and they either approve without reading or stop shipping. One team of six saw median time to merge go from about four hours to fourteen when its weekly pull requests doubled. With one team lead and many developers producing changes, review is a fight you lose before it starts.

What helps is writing down what you’re building and its hard constraints, in documentation and in code. That’s not quite spec-driven development, which works through a spec for one change. It’s describing the system so an agent can check its own work against guardrails. Few organizations have that maturity, and in software you often don’t know where you’ll end up. When you can’t say what the system should be, say what it must not be: the database is never exposed to the public internet, and the industry’s standards apply. The same negative constraints serve as input to the agent and as a review checklist afterwards.

Tests are the obvious deterministic check, but they come with a trap. Give an agent the rules, let it write the tests, and at some point it starts fitting the tests to its own code. Everything passes, and you find the holes later. Newer models may do this less, but we haven’t verified that. Then there’s the loop where one agent writes the pull request, the reviewer asks another agent to review it, and a human sits in between pressing approve. That’s AI reviewing AI with a human signature on top.

Anthropic’s answer, for its own team, is to review intent rather than every line, and to move testing to after the merge. Most companies can’t run that experiment, but the intent half points somewhere useful.

Review against intent

This is where the transcript idea pays off.

How do you review agent pull requests without reading every line?

Capture what the person intended. Then have the review agent and the human check the change against that intent.

Today, intent shows up in commit messages and pull request descriptions, or it doesn’t. A commit message written by a person is condensed to a few words. One written by an agent is a wall of text: left unconfigured, Claude and Codex produce War and Peace. One script we’ve seen measured pull requests where comments made up more than half the change. That isn’t slop exactly. It’s bloat. And it compounds: the next model reads the repo, sees the pattern and writes even more, and every later session pays tokens to read through it and can get confused by it.

Once it matures, intent capture of the kind Entire is building could fix the input side. With a clean record of intent, minus secrets and noise, the review agent checks the diff against what was asked for. The human reviews the intent itself: is this what you actually wanted? One team lead can keep up with short statements of intent. Nobody can keep up with every line of fifty agent pull requests a day.

Until that exists, the tools you already have get you part of the way. Cap pull request size, make agents write a short “what I asked for and what I verified” block, and set writing rules so agents produce less text. We covered those writing rules in episode #22.

Where the new hosts stand now

When we last looked at this in July, Origin had only a waitlist and Entire’s mirror network was nine days old. Here are two corrections and some updates.

Is it time to move to Cursor Origin or Entire?

Mirror if you’re curious, but don’t move a working GitHub setup with real automation yet.

The 22.6 commits a second we quoted for Origin came from launch coverage, and we couldn’t find it in Cursor’s own material. Cursor’s attributable figures are the push numbers above. Origin has been in early beta on paid Cursor plans since August 17, the same day GitHub had a 7-hour, 47-minute outage. Mirrored repos keep GitHub as the source of truth, and for now Origin’s CI apps work only on repos hosted on Origin. Cursor was acquired by SpaceX in August, which some buyers will treat as a procurement question.

Entire now accepts pushes to entire/unmirrored/ branches while GitHub is down. Its docs say those branches aren’t backed up or copied to other regions yet. Entire also documents native repositories that don’t depend on GitHub, and new accounts no longer require a GitHub login.

GitHub’s stacked pull requests became generally available on October 6. On GitLab.com, new rate limits for Free and unauthenticated use are scheduled to take effect on October 19, and Premium and Ultimate follow in January 2027. Per GitLab’s rate-limit documentation, each authenticated Git clone, fetch or push over HTTPS counts as one request, alongside API and web requests. Unauthenticated Git over HTTPS is exempt from the new anonymous allowance. The docs still label the limits as proposed, so treat them as scheduled rather than enforced. Either way, a host can rebuild for more agent traffic and still meter yours.

What this means for teams

For most teams, Git isn’t the bottleneck. Credentials, the review process and the API budget are.

• Lock down what agents can leak. Replace one static credential your agents can read with a short-lived one, starting with whatever sits in a .env file. Before you store transcripts, decide who can read them; with Entire, consider a separate checkpoint repository with tighter access than the code.
• Find one autonomous job that clones a full repo to read a few files, and point it at the file API instead.
• Add an intent block to your pull request template: what was asked for and what was verified. Have your review agent check the diff against it.
• Count your agents’ requests on GitLab.com. Add up API calls and authenticated Git-over-HTTPS clones, fetches and pushes per hour against the new limits: before October 19 on Free or unauthenticated use, before January 2027 on Premium or Ultimate.

Common questions, answered

Resources

• Building Git infrastructure for agent-scale development (GitHub Blog): GitHub’s growth numbers, why replicas slow writes, and the move to Azure Blob Storage with ref updates still coordinated. The 35x figure is an internal benchmark, and the post doesn’t say whether it measured merges into one trunk.
• Git at any scale (Cursor Blog): Vicent Martí’s account of Origin’s write-ahead log in S3 and its compare-and-swap ref updates. Use it for Cursor’s attributable throughput figures instead of the demo number.
• Git was built for humans, agents need an upgrade (GitLab Blog): the clone tax, server-side queries, and GitLab’s own caveat on its numbers.
• Redesigning source control at Anthropic: the far end of the same idea, with Anthropic’s own warning that Scion doesn’t suit general use.
• gitretriever (Datadog Engineering): a version of “don’t clone, ask” that a customer built for itself, and evidence that you can get that win without changing hosts.
• Entire security docs: what Entire redacts, what stays local, and why redaction is only a safety net. Read it before you turn on checkpoints in a repo that touches credentials.
• Temporary security credentials in IAM (AWS): how short-lived credentials expire, and why a leaked one stays usable until it does.
• GitLab.com rate limits (GitLab Docs): the proposed limits, which Git transports they count, and the exemption for unauthenticated Git over HTTPS. The announcement has the October and January schedule.