Git at any scale | Hacker News Reader

Git at any scale

cursor.com

371 pointsmeetpateltech116 comments

It’s hard to overstate the reputation of the author of this post. Everything good about GitHub’s internal systems seemed to have his name all over it (I realize that today this statement hits different than a few years ago). Our times at GH didn’t overlap much but hearing the fact that he’s working at cursor increases my estimation of their engineering org by leaps and bounds.

Came here to share a related fact :)

This solution with WAL and compact resembled the database internals as taught by Martin Kleppmann.

I asked Claude about this observation, and learned that the author (Vicent Martí) wasn't only core Github developer but also worked on Vitess from PlanetScale.

So no surprise these two contexts resulted in the design of Continuity as it is.

https://vitess.io/ - Vitess is a database clustering system for horizontal scaling of MySQL.

The biggest, richest companies in the world are hiring top talent. Who woulda guessed.

We're talking about MSFT here. I think they can afford whatever Cursor can. They're just not taking reliability seriously.

It’s a different game. If you’ve got tonnes of employees you have to think about how all the other employees view comp fairness.

Every large co runs into this problem. New cos have fewer obstacles to paying more… until they get big.

If he has such a good reputation, why doesn't he write well? He never actually introduces the point of the article, he merely alludes to it in the concluding paragraphs.

Seems well written to me.

thanks for the comment ! will read that now

however whether or not this improves their team - who knows! team dynamics is a hard thing to achieve, not just throwing good people in the same bucket ..

Author or not - I just get a shitty machine translation of the original post into German. This is so annoying, machine translated stuff is much harder to read than the original English.

Oh, and yes the preferred language in my browser is set to English.

> What about consensus? Elections? Which server is the primary for a given repository? It also doesn't matter! There's no state and no consensus here. Any server can be the primary. All updates to the write-ahead log are synchronized with an atomic compare-and-swap (CAS) operation on S3, so it's always safe for any instance of a repository to receive a push.

Again reminded of what an amaizing piece of engineering S3 is (99.999999999% - 11 nines of durability) [1]

1: https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDu...

Not just durability. But read after write guarantee on a distributed object store.

How many startups and corporations built on top of S3 semantics?

Snowflake and the big lakehouses are one of them. They use S3 as giant disk in the sky.

It is amazing and keeps adding new features that make it better. The CAS stuff is absolutely wonderful and only a recent addition!

I really enjoyed the article. What's great about the design outlined in it is that it builds on things that work, like leveraging S3 rather than improvising your own distributed storage system.

Of course S3 goes beyond just Amazon these days. It's a proven design and architecture for doing object storage that you find across different cloud providers that you can also self host if you need to.

This article will no doubt inspire people working on projects like forgejo, gitea, tangled, etc. that are all facing the same scaling challenges.

There's nothing Cursor can do that GitHub/Microsoft can't in 2026... And vice versa... after several years of Cursor vibecoding a GitHub clone while catching up with GitHub's new features. Git's performance it's not the issue; it's GitHub Actions, PRs, etc. The moment Cursors steals paid GitHub customers and gets the same workloads, they will start having similar issues! Stealing engineers who couldn't fix this at GitHub won't make much of a difference. GitHub is not just source control; everybody can set up Gitolite and have an amazingly configurable and secure Git setup. I did this 10 years ago, and CODEOWNERS, protected branches, and tags can't do 1% of what I had done before. Cursor is going all over the place as it can't compete with their model. So, this is a defeat, a pivot, not something glorious!

I'd argue that for whatever other faults they have, Composer 2.5 is pretty competitive as an implement-planned-work model. It's several times faster than Sonnet 5, cheaper, and performance is comparable.

Ears wide open for a model that does better for the same parameters. Pareto efficiency is important, but Anthropic doesn't seem to care about it. They desperately need a Haiku 5, IMO.

I use Claude with the Codex plugin as a critic (sometimes I switch and manually review with Claude). With the Superpowers skill (which I am trying to get rid of), it designs expensive models and implementations, uses cheaper ones for other, less important stuff, including Haiku, sometimes upgrading if it hits a roadblock, and then, finally, reviews with expensive ones. Why would I need Composer unless I'm a poor hobbyist? I care about value created, not about the affordable costs. So far, Claude and Codex have given me a huge ROI and are worth every penny. I don't even use an IDE, although I pay for Zed, or the terminal anymore - Claude and ChatGPT desktop apps give me everything and keep things simple. I just watched Theo yesterday [0], and I realized I'm not alone.

[0]: https://www.youtube.com/watch?v=dLhcLqoff6k

Perhaps you didn't read the article, but the implementation they describe is interesting and perhaps easier implemented in a new project from scratch.

It can't be any worse than current github, and since we've heard plenty of people express their dissatisfaction with github recently, this is actually a good direction for Cursor. A product people may want. We'll see.

> It can't be any worse than current github

Yeah, and the grass is always greener somewhere else.

The only way to finally end up in a happy state about your git hosting and automated workflows running, is by running the whole shebang yourself. Saves you so much sanity in the long-term, even if starting and scaling is slower.

I am just too old, you know. I remember Bitbucket, GitLab, and the plethora of other GitHub killers. GitHub is an ecosystem, not a product. Cursor needs at least a decade to beat that! Maybe more!

Maybe what we need is actually fewer ecosystems, and more products.

People are lazy, and when there are no standards (or when they are not followed strictly), it's no small challenge to build a generic interface that works with multiple providers transparently and interchangeably. Many moons ago, we built an LOS (Loan Origination System) for subprime mortgage companies, and I designed a service that standardized input and output for dozens of mortgage vendors - credit, title - you name it. Every vendor was adding their flavor to the standards; everybody read the standards wrongly to some extent, etc. So, when we were pitching the LOS to a bunch of lenders, many of them came back to us and asked, "Can we just buy your servicing subsystem from you?" although it wasn't a separate product at the time. So, naturally, software engineers don't like one-offs unless they support one of the vendors only (pun intended) - the GOAT. It's not so easy to create or extend a standard that works across all vendor implementations. So, it's human nature to try to avoid this process and just support GitHub. The bigger the mass, the stronger the gravity. The biggest mass has the strongest gravity.

Oh, I told my son to create an org yesterday with Cursor Origin, just in case in 10 years it becomes a thing, and he said: "Oops! Paid customers only!" A GitHub "alternative," really? Well, this is reminding me to cancel my Cursor subscription. I am paying for it but not using it because I don't need it as it offers me nothing. I won't even say that somebody already got my GitHub org name, so I'm sure there will be an aftermarket for org usernames and people who want to get 10-15-20x of their $20 investment. Cursor's shortsighted decision is definitely welcomed by parasites! They rushed out their MVP so badly that they created an identity fiasco!

There’s a trend of doing impressive things by pushing many of the hard problems into S3 and assuming S3 “just works”, then not bothering to explain how S3 works. I guess we do the same thing all the time with other miraculous solutions to hard problems, like assuming that superscalar out-of-order processors “just work”, but in cases like this it makes for an unsatisfying explanation. Especially because S3 is a proprietary product, not an algorithm.

at this point, s3 just means object storage and doesnt mean it actually has to be hosted on aws. theres plenty of other companies that provide s3 compatible storage apis.

APIs, sure, but do they provide the semi-magical consistency, atomicity, durability, and latency guarantees that this depends on?

Yes a good object storage service provider would. It all depends on what you are optimizing for. If you are only optimizing for cost then there are several cheap options that probably don't make sense for hot or warm storage. But if you care about availability, consistency, and latency guarantees then there are different service providers available.

It depends?

Last time I looked many of the alternatives didn't have as good support for some of the transaction-friendly things like conditional writes.

Hopefully if you're trying to build something complex on top of S3 and have any expectation of potentially needing to run on top of other infra in the future, you know which magic features you need and can see which of them are supported by what vendors/OSS projects. :)

But it's not really that different than doing cool stuff by pushing a bunch of things into Postgres or Kafka or [other thing] and not explaining how those things work.

you forgot the bandwidth pricing lile its 2006!

The product is proprietary, but the interface and semantics are documented and freely available. "S3-compatible object store" is a commodity at this point, so it makes sense to treat it as a primitive building block for a distributed system. Of course, each implementation will have its own set of idiosyncrasies and limitations, so you'll still have to make sure that any such divergent behavior doesn't interfere with your own system requirements.

Less of a commodity at this point than a few years ago, what with Minio immolating itself to be reborn as, excuse my French, AIStor.

Which is not to say that TFA’s approach is invalid or uninteresting, but the mechanics of how “an S3” works are really obscure to those of us not in the know.

If you're looking specifically for self hosted S3-compatible object stores, then yes, the selection of available options is smaller. But an object store's ability to self-host is orthogonal to its compatibility with S3's interface and semantics.

I'd argue that S3-compatible isn't exactly what you'd need though. You'd need S3-compatible with similar latency characteristics for read and write, with four nines of availability and 10+ nines for durability. [0]. Their metrics are, quite honestly, absolutely impossible for most engineering teams and companies to ever achieve, no matter how hard they work at it. In essence, what I mean is Cursor could migrate Origin out of Amazon's S3, but they'd lose a lot of reliability and/or performance in the process.

[0]: https://aws.amazon.com/s3/ (Click on Durability and availability)

This was the thing that caught my attention. It does feel like a lot of "scaling git was hard and complicated, so we're just offloading a significant to a different layer"

I've got no dog in this fight and no specific rationale, but it smells like a shortcut that's going to make some assumptions that might surprise later (maybe not at the global level, but a local level)

Are there any in-depth technical write-ups on S3 that you would recommend reading on S3 to anyone who _does_ want to get into that abstraction?

https://cursor.com/en-US/blog/git-at-any-scale to avoid automatic translations (couldn't find a button to switch)

Thank you so much, what an obnoxious behavior!

The language selection is in the footer. Interestingly, the URL for English is the original post without en-US.

I have to say, when I first heard about Origin, I was skeptical. But now I do believe that the team behind it is capable of pulling it off and shipping something truly incredible.

This is one of the best technical articles I’ve read. And I came away a lot more knowledgable about a technology I have used daily for over a decade and consider myself adept at. Kudos to the author.

It wasn't that long ago that S3 didn't have read-after-write consistency. Crazy to see the explosion in cool stuff built on top after that change. This is a really well written article and definitely changes my perception about Cursor's capability to host a reliable Github replacement. But I could never bring myself to use it because of whose leadership they're under now. It's such a shame.

The key part: they decided to use the unchanged git codebase as their building block. Likely assuming that the people who wrote the Linux kernel probably knew what they were doing. That explains some features that look bizarre from the generic distributed-database perspective. I think the author has had this argument many times, because he reiterates it several times in the post. The rest follows from it.

This was what stood out to me, too! And this:

> Packfiles are the fundamental building block of Git storage and Git networking.

(emphasis mine)

> His approach was storing the objects in a distributed hash table. This was only possible thanks to JGit, a custom Git implementation in Java. Like any good ol' Java library, JGit provides enough interfaces and factories and interface factories to abstract all the details of a normal Git repository, including replacing its on-disk packfiles with a DHT. Although the system worked and results were good enough for normal Git operations, the limitations of the Git protocol (which again, require packfiles to be sent over the network regardless of how you store data on the server) made the git clone performance bad enough to discard the design altogether.

Looks like Java enterprise design patterns aren't all bad after all :-) and the git ecosystem would have profited from a bit of abstraction and separation of concerns here, where network protocol, git domain model, and storage layer are evolved somewhat independently. The domain model is what everyone in the ecosystem needs to agree on, the network protocol is what at least the given local & remote host need to agree on, but storage is mainly a local concern.

Of course the question is whether git would have today's market and mind share if they had gone down that path. The ecosystem would be a lot more heterogeneous, evolving network protocol would probably take much longer, etc.

The problem is that Musk acquired Cursor and therefore Cursor has zero credibility.

I don't care about what they say because of who they sold out to

Because politics?

That was a very well written and interesting article. I really enjoyed it. Some good background for people telling GitHub to "just scale".

GitHub issues are mostly with PRs, CI, etc, not with git

Source?

When I look at one of the recent blog posts (https://github.blog/news-insights/company-news/an-update-on-...) I see crazy charts of PR merged, Commits, Repositories created. These are all git operations and not just some slow Rails app.

Have you read the article?

> Short term, we had to resolve a variety of bottlenecks that appeared faster than expected from moving webhooks to a different backend (out of MySQL), redesigning user session cache to redoing authentication and authorization flows to substantially reduce database load. We also leveraged our migration to Azure to stand up a lot more compute.

The scaling issues they have don’t have git as the bottleneck from what I understand

In the same article it says:

> The number of repositories on GitHub is growing faster than ever, but a much harder scaling challenge is the rise of large monorepos. For the last three months, we’ve been investing heavily in response to this trend both within git system and in the pull request experience.

They also mention large monorepos and git read operations as a core issue in the other incident post that's on the HN front page right now:

https://github.blog/news-insights/company-news/the-august-17...

Very cool write up. But I do have to wonder, is Git the right solution for version control given where we are heading? The commands are all muscle memory for me now at this point, but all the projects I've been working on lately have just all moved to trunk based development, rebasing, and squashing merging feature branches onto main.

I've been working on another project where I have binary files checked in, assets that move or are moved and then changed that git "loses track of" due to the significant change. This just may be me doing silly things like moving a file and then editing it and then committing it and git thinks I deleted it and created a new file.

Fantastic write up, but I still have doubts. As the author states: Serving git at scale is hard. I can only concur.

I feel like the article is a bit light on the design of the WAL and maintenance operations, but maybe it's me.

Anyway, I'd love to have a friendly chat with the author.

Reading this, I can't come to a conclusion other than "git rapidly starts sucking balls for use cases outside of Linux development". Github started with ze_current_fastness but then ended up recurrently smashing into walls because of it.

They should have just failed to scale git and leave place for the less idiosyncratic bazaar or mercurial to take the lead. These were slower, but would probably have scaled much better in the long term.

Github's core innovation was re-centralizing a decentralized SCM (as per article) by giving it nice web UI and features such as PRs. Oh and also free code hosting space. None of which relied on git's peculiarities.

> The "fan-out" is synchronized with a classic consensus algorithm called 3PC (three-phase commit) so that a push is only accepted if a majority of the nodes acknowledge it.

Doesn't 3PC require all nodes to agree, not just a majority?

I would think so - outside this sentence the author indicates they're talking about consensus across all members.

IE - the interactive diagram and note about latency being bounded by the slowest replica

hmmm, I think they might be using a quorum-based 3PC variant. Reading <https://github.blog/engineering/infrastructure/building-resi...> does make it seem like they only need a majority to commit instead of all nodes.

Ah, and then it looks like the losing nodes get marked unhealthy:

> In essence, every write operation goes through a voting protocol, and any replicas on the losing side of the vote are marked as unhealthy—unavailable for reads or writes—until they can be repaired. Repairs are automatic and quick. Because a majority agreed either to accept or to roll back the update, there are still at least two replicas available to continue accepting both reads and writes while the unhealthy replica is repaired.

But not sure how unhealthy nodes are discovered. Maybe the coordinator marks them, and if the coordinator crashes it allows stale reads and detects invalid writes?

Took me some thinking to figure out where this design is not optimal.

The first step is to separate the problems. The content addressable object storage and storing the refs. libgit2 has two backends for those already.

Then use appropriate distributed dbs for the two backends. With large scale something like Cassandra for the objects (AP) and FoundationDB (CP) for the refs (or the equivalent Amazon services).

For performance the objects need to be cached locally and requests routed to nodes where the objects are cached locally.

Part of the problem is that the API you're supposed to provide externally depends on git packs which you might have to reconstruct on the fly from such a system.

Exactly. I would love to see that design work, but it doesn't for any realistic workload.

Then the local cache system needs to provide or cache those git packs

I don’t see a button to switch to English and the German text is a pain to read, comparable to Reddit auto translate..

The button is apparently in the footer: Another comment earlier has the english-only link: https://news.ycombinator.com/item?id=49376960

Great read - the main takeaway I had was “Spokes has an insane replication protocol for a database (Git) with such low write rates”. Leader/follower + LSN Binlog, WAL, whatever - just not their distributed systems nightmare coordination system

GitHub’s scaling problem is handling huge numbers of relatively small repos at high concurrency. This is not the same problem as Google, Microsoft, Meta et al who have terabyte-scale monorepos, and all have lazy-load solutions to create virtual clones where only the parts you are working on are loaded onto local disk. Microsoft Windows moved to GitHub some time ago and they used the filesystem approach OP dismisses too blithely:

https://github.com/microsoft/scalar

Any ideas how these interactive animated diagrams were made?

Cursor's entire schtick is providing AI coding tools. I'll give you one guess as to what they probably used for the diagrams.

Looks like a React component tree binding to SVG nodes. Right click and inspect element. Right click on the svg element in the (chrome) inspector. Choose `break on` > `subtree modifications`. That led me to look through the network tab for this: https://cursor.com/marketing-static/_next/static/chunks/0sqf... (look for the text "GitObject" in this file)

Cursor Origin Code Hosting

https://news.ycombinator.com/item?id=49334209

how does git lock the repo on extreme concurrency? let's say 1000 devs writing to the same repository? eventually you'll have saturation of the resource locking everyone out.

When I've worked at big companies they had bespoke SCM that resolved this. But i'm curious about mid size teams that are big enough to overwhelm git but not so that they can rewrite it.

If those 1000 devs write to different branches then you only take a light lock on the branch, usually that is not the problem.

The problems start when there is a lot of changes on one branch and usually the first band-aid is to put a commit queue in place. The few very big companies that need even more than that probably aren't going to put their code on a new git hosting provider tomorrow.

that sounds like a practical solution once you reach that size. yeah you're right i think only the refs are locked, the rest of the objects can all be submitted concurrently, so it would be high contention refs/branches that would be a pain point.

This is an excellent post that explains why their version of git hosting is more scalable. But it seems like most people won't need it, assuming Github remains free to use and they get their availability problems fixed.

Will free Git hosting continue to be sustainable, though, without architectural improvements like this?

Free git is unsustainable due to AI

I can't disagree with this in the current state.

But I believe it would be easy to fix with limitation for the free accounts (CI time limit, total size of repo limit etc.)

I don't think vibe coders care for CI limit. And if you put repo limit they can just tell Claude "make sure to optimize for restrictions"

Agents didn’t just make Git harder to host. They likely mandate a new layer between humans and code, and that layer now needs its own versioned artifacts.

A few scaling trends, all triggered by coding agents, make it unsustainable to keep stacking agents on the interfaces designed for human uses — Git, PRs, CI, review:

1. Time to produce a given feature dropped by orders of magnitude. Review, CI, and branch workflows were calibrated to human writing speed. The gates did not get faster.

2. Users will not consume 100x more features. They expect 100x more value from each interaction. The interesting artifact is no longer “the code that implements a feature.” It’s the mapping from intent to outcome.

3. Intermediate work exploded while durable output shrank. Agents leave scratch repos, traces, failed attempts, discarded implementations. The thing the user actually wanted got smaller and more valuable — closer to “satisfy this intent” than “ship this feature.” That is a lure to build much more complex software, and we are versioning the scrap heap with the same tool we use for the product.

4. The primary goal of software engineering till today is to share actions between users, so their intent are served through the same routines. Such software only satisfy commonality. It's now possible to build based on peculiarity. With that, a whole new space of personalization.

This looks like the history of machine tools: first powerful and imprecise, then more powerful and enormously precise. Software has to make the same jump — much more useful, and much more personalized.

I don’t know what the end state looks like. I doubt Git should remain the central mechanism.

Edit: Revised with help of AI.

> The productivity cost of five minutes of downtime in your CI system is hard to quantify in dollars, but it is, by any measure, a humongous amount.

Not sure the last month GH had fewer than five minutes of downtime - but I think it's been a while

It doesn't matter how many technical blog posts they put out, I will still never host my code with Elon Musk.

Same. Sometimes you have to live with inconvenience to keep a moral stand. Those down-voting don't understand why any sacrifice of convenience or tech is worth it.

I agree, although I don't think their offering is even convenient. I also don't use Cursor (I think most people have moved on to Claude Code/Codex), so also not helpful from that point of view. Regardless, not only do I find it morally wrong to give Musk money, I think it's a bad business idea. He can't be trusted and your code is at risk on his platform.

> Those down-voting don't understand why any sacrifice of convenience or tech is worth it.

FWIW my comment was actually highly upvoted but it looks like X employees and/or Musk fans showed up and downvoted all criticism in this thread.

I missed the news about SpaceX acquiring Cursor for 60 billion. Good to know.

Agreed. The recent privacy kerfuffle with Grok CLI (plus the untrue "it's your fault" response, although they then changed the behavior) is a really bad sign for how your code and data will be the treated.

I really wish someone would make a version of the opencode desktop application that's as responsive and well-made as Cursor so I can promptly move over.

It's time to look back on S3's 20-year history

https://blog.fnil.net/s3-at-twenty/

I left Cursor after Mr Musk acquired it and began pushing Grok.

Why go back to a centralized model at all. Guess which project isn’t suffering in this whole debacle of GitHub? The one git was made for of course.

Just imagine the deluge of inbound sales pitches Cursor is now getting from every S3 vendor out there.

FTP mirrors is all you need to "scale".

Unfortunately, Git is not as distributed as it claims to be.

WAL on S3 is the new normal.

Interesting, wonder how Cursor will scale with Git with this approach.

I would parse it as event sourcing at its finest + cqrs

Remember, don't put large objects in cnt, it makes it a real pain later.

cnt?

Continuity system of Cursor. Brilliant name, just like the octo"cat"! Elon will approve.

Can't edit: Looks like they edited this page after the fact[1]: It directly said:

> Continuity (cnt for short) is the git storage system [...]

I guess someone realized that turned this into even more of a joke than it already is.

[1]: https://web.archive.org/web/20260818210312/https://cursor.co...

Reply on news.ycombinator.com