Make your monorepo feel small with Git’s sparse index
github.blog
github.blog
I spent a good chunk of time working around the lack of sparse indexes in libgit2, which produced speedups on the order of 500x for certain operations, because reading and writing the entire index is unnecessary for most users of a monorepo: https://github.com/libgit2/libgit2/issues/6036. I'm excited to see sparse indexes make their way into Git proper.
Shameless plug: I'm working on improving monorepo-scale Git tooling at https://github.com/arxanas/git-branchless, such as with in-memory rebases: https://blog.waleedkhan.name/in-memory-rebases/. Try it out if you work in a Git monorepo.
I'm intrigued by this but the readme could maybe use some work to describe how you envision it being used day-to-day? All the examples seem to be about using it to fix things but I'm not at all clear how it helps enable a new workflow.
Even if it was just a link to a similar tool?
There's a decent discussion here on "stacked changes": https://docs.graphite.dev/getting-started/why-use-stacked-ch..., with references to other articles. This workflow is sometimes called development via "patch stack" or "stacked diffs". But that's just a part of the workflow which git-branchless enables.
The most similar tool would be Mercurial as used at large companies (and in fact, `git-branchless` is, for now, just trying to get to feature parity with it). But I don't know if the feature set which engineers rely on is documented anywhere publicly.
I use git-branchless 1) simply to scale to a monorepo, because `git move` is a lot faster than `git rebase`, and 2) to do highly speculative work and jump between many different approaches to the same problem (a kind of breadth-first search). I always had this problem with Git where I wanted to make many speculative changes, but branch and stash management got in the way. (For example, it's hard to update a commit which is a common ancestor of two or more branches. `git move` solves this.) The branchless workflow lets me be more nimble and update the commit graph more deftly, so that I can do experimental work much more easily.
I think the monorepo workflows which Git isn't good at are things like branching, code review, and feature/version management. But there's no reason that Git should have to be slow just because a repository is large. It's more things like "merge commits don't scale in a monorepo", which I would agree with, but that's not related to the performance of the index data structure.
I have repository https://example.com/myrepo
And i can simply do:
svn co https://example.com/myrepo/some/directory/
And i can work with that subdirectory as if it was actual repo. Completely transparently.
This i really miss in git.
The tests are a trickier problem, in theory you could lock version numbers (the new behavior increments to the new tests) but tests have bugs, and this doesn’t solve two people making behavioral changes at the same time.
Short of merging the tests into the respective projects, going full feature toggle for all breaking changes is the only other lever that comes to mind.
If your form html is reasonably semantically structured, and/or you give meaningful IDs to each interactive element, then your tests can be made somewhat robust against content changes. E.g. "click button #next-step" rather than "click the word Next".
If your frontend is a desktop or mobile app, look for ways to introspect the widget hierarchy to do the same thing (I know this is possible in e.g. Swing).
$ git clone --depth=1 <url>
$ cd repo
$ git fetch --unshallow &
$ <do work>At places I’ve worked when we thought about canary testing we ignored the fact that there were multiple versions of the software running in parallel, we classified it as part of an orchestration process and not a reality about the code or env, but we really did have multiple versions of a service running at once, sometimes for days.
Similarly if you’ve got a setup where you can upgrade users/regions/etc piecemeal (opt-in or by other selection criteria) I don’t know how you reflect this in a monorepo. I’m curious how Google actually does this as I recall they have offered user opt-in upgrades for Gmail. My suspicion is this gets solved with something like directories ./v2/ and ./v3/ — but that’s far from ideal.
The "opt-in upgrades" are all live code. I know more than a few "foo" "foo2" directories, but I wouldn't want an actively-delivered, long-running service to be living in a feature branch so I would still expect anyone to be using a similar naming scheme.
youd just need to tag your releases, right?
In large monorepos the supposition is you've got a class of compatible apps and services bundled together. Version dependencies are somewhat implicit: the correct version for each project to interoperate is whatever was checked-in together in your commit.
I don't know how it works in practice at different orgs, but there's certainly an idea I've heard repeated that you can essentially test, build, and deploy your monorepo atomically, but the reality in my experience is you can't escape the need to think about compatibility across multiple versions of services once you use techniques like canary testing or targeted upgrades.
Again, this fundamentally misunderstands the purpose of the source code repo and how it relates to the build artifacts deployed in production. If you are waiting for something to happen in production before landing some change, that tells me right there you have committed some kind of serious error.
Its very common to need to wait for some version of a binary to be live before updating some associated configuration to enable a feature in that binary (since the dynamic configuration isn't usually versioned with the binary). It's possible that some systems exist that fail quiet, with a non-existent configuration option being silently ignored, but the ones I know of don't do that.
If I'm waiting on an update in a 3rd party library... you have to wait for the 3rd party library to ship. Different binaries are effectively that.
So, yea, I could also feature flag everything, flip the flag when the conditions are right, but sometimes its easier /simpler to just wait to merge.
Also, you wouldn’t expect a schema change to occur with code that requires it. Those will need to happen earlier.
Real systems are complex. A monorepo is one attempt at capping the complexity to known permutations. For smaller teams, it might collapse to a single one.
Branches are cut for releases. Binaries are versioned.
Your understanding of the motivations for monorepo is flawed. I've never heard anyone even advocate for this as a reason for monorepos. For some actual reasons people use monorepos, see https://danluu.com/monorepo/
Regarding your question, which I re-emphasize has got nothing to do with the arrangement of the source code, the solution is to simply treat your protocol as API and follow these rules: 1) Follow "Postel's Law", accepting to the best of your abilities unknown messages and fields; 2) never change the meaning of anything in an existing protocol. Do not change between incompatible types, or change an optional item to required; 3) Never re-use a retired aspect of the protocol with a different meaning; 4) Generally do not make any incompatible change to an existing protocol. If you must change the semantics, then you are making a new protocol, not changing the old one,
> I don’t know how you reflect this in a monorepo
You don't. Why would the deployed version of some application be coded in your repo? It's simply a fact on the ground and there's no reason for that to appear in source control.
As soon as you discuss network traffic (or really anything that crosses an RPC boundary), things get more complicated, but none of that has anything to do with a monorepo, and monorepos still sometimes simplify things.
So there's a few tools that are common: feature flags, 3 stage-rollouts, and probably more that are relevant, but let's dive into those first two.
Feature "flags" are often dynamically scoped and runtime-modifiable. You can change a feature flag via an RPC, without restarting the binary running. This is done by having something along the lines of
if (condition_that_enables_feature()) {
do_feature_thing()
} else {
do_old_thing()
}
Then you have "3-phase" updates. In this process, you have a client and server. You want to update them to use "v2" of some api, that's totally incompatible with v1. You start by updating the server to accept requests in either v1 or v2 format. That's stage one. Then you update the clients to sent requests in v2 format. That's stage two. Then you remove all support for v1. That's stage three.
When you canary a new version of a binary, you'll have the old version that only supports v1, and the canary version that supports v1 and v2. If it's the server, none of the clients use v2 yet, so this is fine. If it's the client, you've already updated the server to support v2, so it works fine.
Note again that all of this happens whether or not you use a monorepo.
But the supposed benefit of the monorepo was that you could update everything from v1 to v2 in one atomic commit, and that hasn't actually happened. You have to make the changes in exactly the same way as if your client and server were in separate repositories; indeed it may be harder to get your integration tests to test every combination that will actually be deployed. So when there's a network boundary in between - or more to the point a deployment boundary in between - you end up with the costs of the monorepo but not the benefit.
No, a benefit of a monorepo is that you can update your build-time dependencies atomically. No one has claimed that a monorepo lets you magically do atomic RPC API migrations.
> You have to make the changes in exactly the same way as if your client and server were in separate repositories; indeed it may be harder to get your integration tests to test every combination that will actually be deployed.
It won't though: if you cover both code paths at each stage of the three stages, you'll cover all possibilities (its also not clear that you need integration tests to cover this, you can fake the client or server and test against both versions and be fine). This is the same if you've got multi or mono-repo, although with a monorepo there's a decent argument that maintaining integration tests is easier, as you can build both the client and server from within the repo.
> So when there's a network boundary in between - or more to the point a deployment boundary in between - you end up with the costs of the monorepo but not the benefit.
This presumes that there are some costs. I'd counter that there's no real difference in this situation, and so when choosing mono- vs. multi-repo, this kind of situation shouldn't influence your decision, because it doesn't matter. You should base your decision on the cases where the choice has an impact.
Plenty of people have claimed and continue to claim that a monorepo lets you magically do atomic API migrations, and skate over how the fact that many APIs go over RPC these days.
> It won't though: if you cover both code paths at each stage of the three stages, you'll cover all possibilities (its also not clear that you need integration tests to cover this, you can fake the client or server and test against both versions and be fine). This is the same if you've got multi or mono-repo, although with a monorepo there's a decent argument that maintaining integration tests is easier, as you can build both the client and server from within the repo.
This relies on assuming that you perfectly maintain the previous codepath as-is, otherwise your tests won't cover what's actually going to run. In a multirepo setup the idea that you run your integration tests against version X of server code and version Y of client code is already something you're conscious of (and you absolutely do need to integration test your real client with your real server, otherwise you won't uncover your false assumptions about the behaviour of one or the other), so testing the new release of the server with the unmodified old release of the client is something you can do very naturally, whereas in a monorepo you have to make a very deliberate effort to test anything other than current client against current server.
Nobody that knows what they are talking about makes this claim. People making this claim are confused and misunderstand. Just because there are a group of people who don't understand the thing they are advocating for, doesn't make the thing any better or worse for whatever purpose they are espousing it for. It just means that they don't know what they are talking about (see also "even a broken clock is right twice a day").
To wit: if I'm using a monorepo to produce my back-end service and the client libraries I ship to my customers, who must update their deployed version themselves, how do these proponents of "magic monorepo atomic RPC rollouts" propose that works exactly? Clearly, it doesn't. Updating your own environments is really no different. All deployments are eventually consistent. C'est la vie.
> whereas in a monorepo you have to make a very deliberate effort
When dealing with API or data model changes, you have to deal with the same, deliberate effort to schedule/stage the release of your changes regardless of approach that is used.
Right, but, frankly, that's the statement that a lot of hype and even adoption of monorepos is being driven by people who don't know what they're talking about, for reasons that are nonsense.
> When dealing with API or data model changes, you have to deal with the same, deliberate effort to schedule/stage the release of your changes regardless of approach that is used.
If you align the repository - particularly, the level of granularity at which you tag and at which you check out - with the level of granularity at which you deploy, that can simplify your working practices and make some important details more visible. If your day-to-day development workflow is that you check out version 506 of "everything", than naturally nudges you to thinking of this global monotonic version which is misleading when it comes to actually deploy things. If it's natural to checkout version 523 of foo and version 416 of bar in your day-to-day work, that can help you align more closely with a world where version 523 of foo and version 416 of bar is what you actually have deployed on your servers.
When you have relatively small services, continuously updated, there's not a need to do the sort of precise cross-version testing you're advocating for, as long as you maintain some level of per-feature forward and backward compatibility across your api surfaces, which is far easier than doing cross-version integration tests for every change.
Put another way, you should decouple yourself from caring about particular versions and think more about which features are supported by what's currently deployed. The question you ask shouldn't be "is version X in production", it's "is feature Foo safe to depend on". Unless something is wrong, feature Foo should work just as well in version X and X+1 and X+2, so caring about precisely which you're using is too low level.
I find testing the new version of the server against the actual version of the client that is deployed in production, or vice versa, extremely valuable for avoiding issues.
I'd argue that there are many more APIs which just don't go over network. With monorepo you can do cross-component refactor very easily, this is especially handy when you are working on a widely-used library.
Imagine that you are working on your language's standard library and you want to fix a small API mis-design.
At each commit, your integration test covers the code path that runs at that CL. So for the client, when you switch from the V1 API to the V2 API, you also swap the integration test. Sure you're no longer testing V1, but that's fine, because the artifact being deployed no longer uses the V1 api.
Since the server needs to support both versions, then you'd probably want, for some period of time, tests exercising both paths, but its not challenging to do that. You don't touch the existing tests, and solely add a new test case exercising the new path.
There's no need for explicit cross-version testing with particular versions, because the way you stage the upgrades ensures all the cross-compatibility you need.
> so testing the new release of the server with the unmodified old release of the client is something you can do very naturally
No, this happens perfectly naturally in the monorepo case. When you upgrade the server to support both APIs, you haven't yet touched the client, so you're naturally testing it on the unmodified old client behavior. I'll reiterate, by doing the upgrade in the three-phase way I mentioned, this isn't a problem, and I'm not sure why you think it is.
But that's a manual, deliberate process. You're effectively recreating what your VCS does for you by having explicit separate codepaths for each API version.
You'd still need to update the server to support both API versions during the transition period. We agree the change can't be made atomically in a monorepo, but it also can't be done atomically in a multi-repo environment.
So, since you have two code paths in prod, you also need them in your tests. It's a manual process either way, and completely independent of your vcs.
Importantly, this is the completely natural way to do every release and deploy; you'd have to go out of your way not to do this. Even if you thought version 6 didn't contain any protocol changes, you'd still deploy it exactly the same way. So you end up testing the real combination of code that's about to get deployed, before you deploy it, every time.
Doing so makes this a less pertinent problem.
I'm surprised that the index is not hierarchical, like tree objects in Git's object storage.
With tree objects (https://git-scm.com/book/en/v2/Git-Internals-Git-Objects#_tr...), each level of the hierarchy is a separate object. So you would only need to load directories that are interesting to you. You could use a single hash compare to determine that two directories are identical without actually recursing into them.
In particular, I can't understand why you would need a full list of all files to create a commit. If your commit is known not to touch certain directories, it should be able to simply refer to the existing tree object without loading or expanding it.
I guess that's what this sparse-index work is doing. I'm just surprised it didn't already work that way.
There's performance-related metadata in the index which isn't in tree objects. For example, the modified-time of a given file exists in its index entry, which can be used to avoid reading the file from disk if it seems to be unmodified. If you have to do a disk lookup to decide whether to read a file from disk, then the overhead is potentially as much as the operation itself.
There's also semantic metadata, such as which stage the file is in (for merge conflict resolution).
It's worth noting that you can turn on the cache tree extension (https://git-scm.com/docs/index-format#_cache_tree) in order to speed up commit operations. It doesn't replace objects in the index with trees, but it does keep ranges of the index cached, if they're known to correspond to a tree.
It's been something on my list to address on the mailing lists for a while, just haven't had time.
(Obviously e.g. an image resizer or something else completely decoupled from your business logic should be a separate service / repo anyway—my point is more along the lines of "If something shares code, shouldn't it share a deployment strategy?")
Although, that said, I can at least see the benefits of the "1 service per team" methodology, where you have a dedicated team that's independently responsible for updating their service. I'm more used to associating "microservices" with the model where a single team is managing 5 or 6 interacting services, and the benefits there seem much smaller.
Different teams can make their own decisions, but as a developer on a team that ran our own SRE, I found it came with many advantages. Specifically, we saw very little downtime, and when outages did occur were very prepared to fix it as we knew the exact state of our services (code, infrastructure, recent changes and deploys.) Additionally, we had very good logging and metrics because we knew what we'd want to have in the event of a problem.
And I'm not sure what you mean "from the ground up." We were able to share a lot of Ansible playbooks, frameworks, and our entire observability stack across all teams.
But I think you may also be missing the rest of my post. This is only one possible advantage. Even if the team doesn't perform their own SRE, these services can be scaled independently - both in terms of infrastructure and codebase - even while sharing code (including things like protocol data structures, auth schemes, etc.)
A service that receives 1 SAML Response from an IdP (identity provider) per day per user may not need the same resources as a dashboard that exposes all SP (service providers) to a user many times a day. And an administration panel for this has still different needs.
Yet, all of these services may communicate with each other.
Monorepos are excellent for microservices.
- You can update the protobuf service graph (all strongly typed) easily and make sure all the code changes are compatible. You still have to release in a sensible order to make sure the APIs are talking in an expected way, but this at least ensures that the code agrees.
- You can address library vulns and upgrades all at once for everything. Everything can get the new gRPC release at the same time. Instead of having app owners be on the hook for this, a central team can manage these important upgrades and provide assistance / pairing for only the most complex situations.
- If you're the one working on a very large library migration, you can rebase daily against the entire fleet of microservices and not manage N-many code changes in N-many repos. This makes huge efforts much easier. Bonus: you can land incrementally across everything.
- If you're the one scoping out one of these "big changes", you can statically find all of the code you'll impact or need to understand. This is such an amazing win. No more hunting for repos and grepping for code in hundreds of undiscovered places.
- Once a vuln is fixed, you can tell all apps to deploy after SHA X to fix VULN Y. This is such an easy thing for app owners to do.
- You can collect common service library code in a central place (eg. internal Guava, auth tools, i18n, etc). Such packages are easy to share and reuse. All of your internal code is "vendored" essentially, but you can choose to depend on only the things you need. A monorepo only feels heavy if you depend on all the things (or your tooling doesn't support git or build operations - you seriously have to staff a monorepo team).
- Other teams can easily discover and read your code.
Monorepos are the best possible way to go as long as you have the tooling to support it. They fall over and become a burden if they're not seriously staffed. When they work, they really work.
For one thing you get clear ownership of deployed code. There isn't one monolithic service that everyone is responsible for babying, every team can baby their own, even if they all share libraries and whatnot.
You also get things like fault isolation and fine grained scaling too.
- Monoliths force you to coordinate deploys, which can be a pain. It's difficult to verify your changes in concert with others.
- Bugs from one team can cause a rollback for everyone.
- One N+1 query, nonlinear algorithm, or GC-contentious endpoint can send the entire system into a spiral and cause non-linear behavior.
- You don't need to scale everything the same (services, databases, etc.) Your authentication pieces need massive scale: 10,000 QPS in an active-active configuration. Your random endpoint for the "XYZ promotion" does 0.5 QPM.
- Engineers that build platforms can operate in Java/Go/Rust, engineers that build product can build in Ruby/Python/Node.
- Payment data growing faster than user data? Scale your payment pieces. Shard their databases. They're safely cordoned off and isolated from anything else. The payments team(s) can focus on this one really hard problem without anyone else knowing or caring at all.
- Engineers can have privileged access to the resources their team consumes and needs. You don't want your customer loyalty team having access to payment team data. Or your appointments team having access to user accounts and authentication.
- You can hire for teams/concerns, and you have an interesting metric: number of services per engineer or team. It's actually pretty useful.
- Mistakes in one corner can sometimes be completely isolated and not take down the entire company. This isn't always the case. Critical infra (auth, service discovery, networking) can take everything out. But your appointments or customer success app won't have any impact to the other business areas. Also, it's great to have those super mission-critical folks focused on only the extremely important function they serve.
- Tech debt can live "under the covers". External teams calling your services might not know about the mess or the ongoing migrations under the hood. They don't need to know. (Granted, if your data model at scale sucks, this doesn't really afford much. But it is an easier puzzle to decompose.)
tl;dr - teams are focused on things they own, scope is controlled, blast radius is limited, pieces scale independently as needed.
So the next thing you decide to do is create some microservice framework that bundles all that stuff in and allow your microservice team to write some business logic on top. But now 99% of your executables are in this microservice framework that everyone is using, and that's the point where a lot of companies go the monorepo route.
Actually most companies do some mix -- have a lot of stuff in a big repo and then other smaller repos alongside that.
That said, there's absolutely no reason to structure your code in a monorepo.
Here's what I think GitHub is doing:
1) Encourage monorepo adoption
2) Build tooling for monorepos
3) Selling tooling to developers stranded in monorepos
Microsoft, which owns GitHub, created the microsoft/git fork linked in the article, and they explain their justification here: https://github.com/microsoft/git#why-is-this-fork-needed
> Well, because Git is a distributed version control system, each Git repository has a copy of all files in the entire history. As large repositories, aka monorepos grow, Git can struggle to manage all that data. As Git commands like status and fetch get slower, developers stop waiting and start switching context. And context switches harm developer productivity.
I believe that Google's brand is so big that it led to this mass cognitive dissonance, which is being exploited by GitHub.
To be clear, here are the two ideas in conflict:
* Git is decentralized and fast, and Google famously doesn't use it.
* Companies want to use "industry standard" tech, and Google is the standard for success.
Now apply those observations to a world where your engineers only use "git".
The result is market demand to misuse git for monorepos, which Microsoft is pouring huge amounts of resources into enabling via GitHub.
It makes great sense that GitHub wants to lean into this. More centralization and being more reliant on GitHub's custom tooling is obviously better for GitHub.
It just so happens that GitHub is building tools to enable monorepos, essentially normalizing their usage.
Then GitHub can sell tools to deal with your enormous monorepo, because your traditional tools will feel slow and worse than GitHub's tools.
In other words, GitHub is propping up the failed monorepo idea as a strategy to get people in the pipeline for things like CodeSpaces: https://github.com/features/codespaces
Because if you have 100 projects and they're all separate, you can do development locally for each and it's fast and sensible. But if all your projects are in one repo, the tools grind to a halt, and suddenly you need to buy a solution that just works to meet your business goals.
The fact that git is very poor at scaling monorepos might mean that it's a bad idea to use git for larger organizations, not that it's a bad idea to use monorepos. If git can be improved to work with monorepos, all the better.
I strongly disagree with that, but I'll let this blog post explain it better than I can: https://medium.com/@mattklein123/monorepos-please-dont-e9a27...
> You keep talking about Google, but a much more famous monorepo is Linux itself.
I thought it was fairly well known that monorepos came directly from Google as part of their SRE strategy. It didn't even come into common usage until around 2017 (according to wikipedia). If I'm remembering correctly, the SRE book recommends it, and that's why it gained popularity.
Also, I don't believe that Linux is a valid interpretation of "monorepo". Linux is a singular product. You can't build the kernel without all of the parts.
A better example would be if there was a "Linus" repo that contained both git and linux. There isn't, and for good reason.
> The fact that git is very poor at scaling monorepos might mean that it's a bad idea to use git for larger organizations, not that it's a bad idea to use monorepos. If git can be improved to work with monorepos, all the better.
Any performance improvement in git is welcome, but anything that sacrifices a full clone of the entire repository is antithetical to decentralization.
The whole point of git is decentralized source code.
I think "monorepo" here is a shorthand for large, complex repos with long histories which git does not scale well to whether or not it is all of the repos for an organization. For example I'd call the Windows OS a monorepo for all of the important reasons.
The "whole point of git" is to provide value to its users. Full decentralization is not necessary for that.
But it’s also larger scale than the vast majority of startups will ever reach. My work has had the same monorepo for 8 years with over 100 employees now and git has had few problems.
That article completely misses the point about project history - a monorepo has the full history of your project, while multiple repos split that history. If you have a good split, everything is fine, but if you're moving code between repos relatively often, everything gets muddled.
It also overstates the need for build artifact management in a monorepo. For 3rd party code that has a good package management solution you can use that, while still using the simpler solution of no dependency management for internal libraries, in the good old C tradition. I will again point to the Linux kernel as a good example of doing this successfully - they don't do any kind of versioned build artifacts for any of the many tens of libraries they use - they just rely on git.
> I thought it was fairly well known that monorepos came directly from Google as part of their SRE strategy. It didn't even come into common usage until around 2017 (according to wikipedia). If I'm remembering correctly, the SRE book recommends it, and that's why it gained popularity.
While Google may have coined this term in 2017, the idea of keeping all of an organization's code into a single repo was around since forever. The ~1k dev company I work for had a single Perforce repo with history going back to 1998 or something like that.
> Also, I don't believe that Linux is a valid interpretation of "monorepo". Linux is a singular product. You can't build the kernel without all of the parts.
This is probably the core of disagreement actually. Linux of course has all sorts of internal parts that can be considered libraries/modules. They famously have a huge amount of drivers, but there are also things like kernel-space implementations of much of the C stdlib, such as kmalloc, compression libraries, a unit testing framework, the ebpf compiler and runtime, more than 40 file systems, network stacks for various protocols, and so many more. Nothing prevents the kernel team from splitting up the kernel into a core 'app' + tens of libraries and add tooling to stitch these together from separate repos (since it's C, you could theoretically do all of this at the single source file level, even). Not to mention, if you want to use a 3rd party library in Linux, there is only one way to do it: you copy its source into the kernel source tree.
Of course, no one is suggesting such a thing, because it's well accepted that the kernel is a 'single product'. But multiple-repo advocates usually miss the fact that everyone starts with a single repo and a single product, and then they gradually isolate parts of that product as 'libraries', and they gradually split up functionalities as 'separate products', and it's not always clear when this separation is actually finished enough to make the decision to split it off into its own repo, if ever.
> Any performance improvement in git is welcome, but anything that sacrifices a full clone of the entire repository is antithetical to decentralization.
> The whole point of git is decentralized source code.
Partial clones are still decentralized source, these concepts are completely orthogonal. As long as I can clone all of the code I am working on and all of its history, it obviously makes no difference if this is a single repo from a multi-repo project or it is a single part of a monorepo.
Also, the whole point of git is managing source code history. The decentralized part is only important for decentralized projects, like Linux. Most projects, either private or public, work in a centralized way, and maintain a central source control server, and local git clones are just nice-to-have caches.
And I want to emphasize again: I am talking about most open source projects here as well, even some of that large ones like Apache. Even GNU typically works in this centralized manner: you get the central repo for some project, you make changes, you rebate your changes on master, you send a patch for review to the mailing list, and if the patch is accepted, you ore some maintainer commit it to the central repo.
In a decentralized workflow like Linux, you start off by cloning one or more relevant authoritative repos (Linus' repo, or Debian's repo etc), you make your changes, you format them as a patch, and you send the patch to the maintainers of the repos you want to change - e.g. Debian's repo for a security fix maybe, or the kernel maintainer for the subsystem you want to change. If they like your patch they take it and put it in their repo. If it's important enough, it will then slowly percolate through the ecosystem in various ways - the maintainer will eventually push it to Linus to be included in an official release, and Debian will eventually take it from Linus' repo. Some mainly not even wait for it to make it to an official Linux release - a cutting-edge distro may directly take changes from another maintainer's repo.
There are extremely few projects that work in this manner.
Edit: added a few more details on just how much stuff is in the Linux source tree, and how much they work exactly like the linked article claims will never scale (no dependency management, no 3rd party artifacts etc).
Hmm -- would a different gitting strategy on the part of the kernel dev team perhaps influence their take on the whole monolithic vs microkernel question? :-)
You most certainly can. Loadable modules have been part of the kernel for over 20 years now.
The fact that many drivers exists out-of-tree should be enough to settle that particular argument.
There are libraries in the kernel perfectly usable on its own. There are also many scripts, analysis and testing tools that live in the kernel that build and runs separately from the kernel itself. Then there's a whole lot of documentation.
Linux is what git what designed for. It sits in a singular respository and no other repositories are needed to build a working product. It can accurately be described as a monorepo, if that particular distinction was important.
Also, there are no external dependencies: if you want a new library to be used, you copy its source to the kernel source tree - a classic monorepo solution, that detractors claim "doesn't scale".
Bullshit. There are very good reasons to use it in some situations. My company is using it and it's a tremendous productivity boon. And Git works perfectly fine for smaller scales.
Obviously, "because Google does it" is a terrible reason. But it's disingenuous to say that's the only reason people are doing it. Not everyone is a moron.
When you do, you have two choices. You can either commit to the monorepo direction and start using non-standard tooling that sacrifices decentralization, or you can break up your repo into smaller manageable repos.
I don't have any problem with small organizations throwing everything into one git repo because it's convenient.
My objection is that when you eventually do hit the limits of git, will you choose to break the fundamentals of git decentralization as a workaround? Or will you break up the repo into a couple of other repos with specific purposes?
I don't like that GitHub makes money by encouraging people to make the wrong choice at that juncture.
I would say the opposite. A lot of companies are fine with independent teams using their own versions of dependencies and their own versions of core code, but at some point that becomes unmanageable and you need to start using a common set of dependencies and the same version of base frameworks to reduce the complexity. That means pushing a patch to a framework means all the teams are upgraded. Monorepos are the most common solution to enforce that behavior.
Look, this is all dealing with the problem of coordination in large teams. Different organizations have different capacities for coordination, and so it's like squeezing a balloon -- yes, you want more agility to pick your own deps but then the cost of that is dealing with so much complexity when you need to push a fix to a commonly used framework or when a CVE is found in a widely used dep and needs to be updated by 1000 different teams all manually.
There is no "right" way. It's just something organizations have to struggle with because it's going to be costly no matter what, and all that matters is what type of cost your org is most easily able to bear. That will decide whether you use a monorepo or a bunch of independent repos, whether you go for microservices or a monolith, and most companies will do some mix of all of the above.
Yes. This is very accurate and also the problem. Monorepos are being used as a political tool to change behavior, but the problem is that it has severe technical implications.
> There is no "right" way.
With git, there is a "wrong" way, and that's not separating your project into different repos. It causes real world technical problems, otherwise we wouldn't have this article posted in the first place.
> It's just something organizations have struggle with because it's going to be costly no matter what, and all that matters is what type of cost your org is most easily able to bear.
It's not a coin toss whether monorepos will have better or worse support from all standard git tooling. It will be worse every time.
The amount of tooling required to enforce dependency upgrades, code styles, security checks, etc across many repos is significantly less than the amount of tooling required to successfully use a monorepo.
This isn’t a useful game to play.
This is completely false. You only need extra tooling on monorepo for gigantic scale. For 90% or maybe even 99% of companies out there using version control, plain git and a "bunch of folders" is all you need, honest to god. This way you can reap most benefits and very few downsides. Detecting changes in folders is pretty easy, as most CIs/CDs have it built-in. Ff not it's easier to program a script than setup multirepo tooling.
Given that Google and Microsoft use monorepos that seems unlikely!
Microsoft forked git and layered their own file system on top of it to support a centralized git workflow so that they could have a monorepo.
If you want a centralized, trunk-based version control, don't use git.
It's funny how each company decides to solve these problems.
Google called in the computer scientists and designed a better centralized vcs for their purposes. Good on them. It'd be great if they open sourced it. So typical of Google to invent their own thing and keep it private.
Microsoft took the most popular vcs (git), and inserted a shim layer to make it compatible with their use case. How expected that Microsoft would build a compatibility shim that attempts to hide complexity from the end user.
Meanwhile, Linux and Git are plugging along just fine, in their own separate repos, even though many people work on both projects.
Yeah like their build system... Bazel, that's completely closed source.
It would be hard to open source if its dependencies only exist on Borg. I think that was a problem with Blaze (now Bazel).
No one should work on monorepos because...
monorepos are bad because...
git can't easily handle them and we shouldn't fix that because...
No one should work on monorepos...
If you think that git needs to be "fixed" or "updated" to support a centralized vcs server to do partial updates over the network, then I think you've missed the point of git.
So, yea, for companies, monorepos are a no brainer in a lot of ways.
For open source, separate repos makes more sense.
To expand on corporate monorepos, if you can still set up access control (e.g., code owners to review additions by domain) and code visibility (so there isn't _unlimited_ code sharing), then I can't think of a reason to not use monorepos.
One of our tasks when building the monorepo was proving it was possible to split it again. It was trivial and we have tools to help us avoid complexity.
We’re not using Github so that part doesn’t apply to me.
Also, nice of you to assume we’ll get to Google scale, but thanks to the monorepo, I was able to make a few pull requests reducing duplication and reducing line count of app by thousands ever since. So I really don’t see us getting into Google scale anytime soon. We're downsizing.
I also find it ironic that you’re accusing people of “copying Google” in a parent post but you’re the one assuming that everyone will hit Google limits…
As a best practice, there's a reason that Linus started git in a separate repo, rather than as part of the Linux project. The reason is that if you put too many projects into one git repo, and it gets too large, you do eventually hit a scale where it becomes a problem.
A very simple way to mitigate that is to keep each project in its own repo, which you can easily do once you start hitting git scale problems.
Thankfully, one of the original git use cases was to decompose huge svn repos into smaller git repos, so the tooling required is already built in.
> I honestly find it ironic that you’re accusing people of “copying google” in a parent post but you’re the one assuming that everyone will hit Google limits…
I think you got the wrong take there. I'm saying that Google's monorepo approach is only valid because they invested so heavily into building custom tooling to handle it. We don't have access to those tools and therefore shouldn't use their monorepo approach.
If you're going to use git, you're going to have the most success using it as intended, which is some logical separation of "one repo per project" where "project" doesn't grow too out of hand. The Linux kernel could be thought of as a large project that git still handles just fine.
Tragically, I think if Google did opensource their internal vcs and monorepo tooling, they would immediately displace git as the dominant vcs and we would regress back to trunk-based development.
Nope. The reason they are in a different repositories is that they are completely different things, with virtually zero code shared between them. There is no point in putting them in a single repository. The teams are different, the process is different, the release cycle is different.
The reason people like me use monorepos is not because we mistakenly believe "each company needs only one repo". It's because there is a tangible and measurable advantage in grouping some things together. That's it.
> If you're going to use git, you're going to have the most success using it as intended, which is some logical separation of "one repo per project" where "project" doesn't grow too out of hand.
No, that's patently false and absurd. I had the experience of using one repo per project and it was a horrible experience full of bureaucracy, and I know other people with the same experience. A git monorepo is working fine for my company and has been for years. I know other companies who also feel the same, and no, the "sky is not falling" as you seem to believe.
Claiming that "there can be only valid method, and it's mine" is bullshit. Engineering is not about silver bullets.
> Tragically, I think if Google did opensource their internal vcs and monorepo tooling, they would immediately displace git as the dominant vcs and we would regress back to trunk-based development.
You keep forgetting that not every company operates at the Google level. Git is more than fine for 90% of the cases out there. There is more to tech than FAANG.
> If you ever do hit a git limit where it's no longer comfortable to keep the whole repo on each developer machine, I would encourage you to split up the repo into separate project-based repos rather than switching to Microsoft's git fork.
Sorry to be blunt, but I'm not taking advice from an internet stranger with opinions grounded purely on dogma, and with an axe to grind. I will carefully study my options and take a decision based on my needs. There are no silver bullets in engineering.
Curious to hear more specifics on this. Did you migrate from separate repos to a monorepo and subsequently measure improved productivity as a result?
The biggest differences were in changes that would break the consumers. For this cases we had to go back and patch the original library, or revert and start from scratch. But even in the easy changes, just the "bureaucracy" of opening tens of pull-requests, watching a few CI pipelines and getting them approved by different code owners was also large.
Now, whenever we have changes in one of the core libraries, we also run full tests in the library consumers. With tests running in parallel, sometimes it takes 20 minutes (instead of 4, 5 hours) to get a patch affecting all frontends tested, approved and merged into the main branch.
Also, everyone agreed that having multiple PRs open is quite stressful.
This can also be done with CI triggers across repos.
Most (all?) of Google OSS software is hosted on either Gerrit or Github. Git is not used by the "google3" monorepo, but it's used by quite a few major projects.
Here's some of the many reasons why monorepos are excellent:
- Continuous integration. Every project is almost always using the lastest code from other projects and libraries it depends on.
- Builds from scratch are very easy and don't need extravagant tooling.
- Problems due to build versions in dependency management are reduced (everyone is expected to use HEAD).
- The whole organization settles on a common build patterns - so if you want to add a new dependency you wouldn't need to struggle with their build system. Conversely, you need to write lesser documentation on how to build your code - cause that's now standard.
- CI breaks all the time. Even one temperamental test from anywhere else in the organization can cause your CI run to fail.
- Building the monorepo locally becomes very complicated, even to just get your little section running. Now all developers need all the tools used in the monorepo.
- Dependencies get upgraded unexpectedly. Tests aren't perfect, so people upgrade dependencies and your code inevitably breaks.
It's cool that everyone is on the same coding style, but that's very much achievable with a shared linter configuration.
The problem is choosing a monorepo because the tooling isn't suited for monorepos.
Trying to build a monorepo with git is like trying to build your CRUD web app frontend in c++.
Sure, you can do it. Webassembly exists and clang can compile to it. I wouldn't recommend it because the tooling doesn't match your actual problem.
Or maybe a better example is that it's like deciding the browser widgets aren't very good, so we'll re-render our own custom widgets with WebGL. Yes, this is all quite possible, and your result might get to some definition of "better", but you're not really solving the problem you had of building a CRUD web app.
Can Microsoft successfully shim git so that it appears like a centralized trunk-based monorepo, the way you'd find at an old cvs/svn/perforce shop? Yes, they did, but they shouldn't have.
My thesis is they're only pushing monorepos because it helps GitHub monetize, and I stand by that.
> Tests should only execute against code that changed. Builds should only build the thing you want to build, not the whole repository.
How do you run your JS monorepo? Did you somehow get bazel to remote cache a webpack build into individual objects, so you're only building the changes? Can this even be done with a modern minimization tool in the pipeline? Is there another web packager that does take a remotely cachable object-based approach?
I don't know enough about JS build systems to make a monorepo work in any sensible way that utilizes caching and minimizes build times. If anything good comes out of the monorepo movement, it will be a forcing function that makes JS transpilers more cacheable.
And all this for what? Trunk-based development? So we can get surprise dependency updates? So that some manager feels good that all the code is in one directory?
The reason Linus invented git in the first place was because decentralized is the best way to build software. He literally stopped work on the kernel for 2 weeks to build the first version of git because the scale by which he could merge code was the superpower that grew Linux.
If you YouTube search for "git linus" you can listen to the original author explain the intent from 14 years ago: https://www.youtube.com/watch?v=4XpnKHJAok8
If this is a topic you're passionate about, I'd encourage you to watch that video, as he addresses why decentralizing is so important and how it makes for healthy software projects. It's also fun to watch old Googlers not "get it".
He was right then and he's right now. It's disappointing to see so much of HN not get it.
Disclosure: I'm the product manager for Git Systems at GitHub, where much of this work is based.
Also, I appreciate that you're upstreaming to core git and not trying to build a technical moat.
But why do so many people want these monorepos?
This seems like a serious developer education problem to me.
If I'm wrong that Google is the cause of this cultural shift toward monorepos, what do you think caused it?
I can't figure out why so many people are married to git but insist on leaning into its weaknesses (large repos) instead of its strengths (smaller composable repos).
In siblings to this thread, we have people reminiscing about the good old days of subversion and asking for git to emulate those features. It's quite frustrating to watch.
I think I've been operating under the illusion that most developers understood git well enough to realize it's a better model than svn. But in reality developers would have been more than happy to use "SvnHub" if that had come along first.
Lots of people elsewhere in the thread have given reasons that I won't repeat, but I will say I think it's a very different situation for an engineering-focused company compared to, for example, open source projects.
In a company you get a lot of advantages from uniformity of tools and practices and dependencies across the many projects/products that the company has, and you have the organizational structure to onboard new employees onto that way of working and to maintain the level of uniformity and actually benefit from it.
In open source, every project and even every contributor is sovereign unto themselves. In this situation, the difficulties of cross-project changes are primarily human organizational challenges - even if you could collect a lot of projects into a monorepo, you couldn't get the big benefits from it because you wouldn't be able to get everyone to agree to stick to a single set of tools and do the other things that make monorepos powerful.
I think monorepos are great for companies.
But using git as a basis for a monorepo system is a bad idea for the reasons you suggest. It's totally the wrong tool for the job.
I remember the transition period when (nearly) everyone in the open source world moved over multiple years from subversion - or in some cases cvs - to git. The advantages of decentralised development for open source were really clear, and so was the way git tracked branch and merge history (when git was first developed, subversion barely tracked merge history at all, which made it easy to end up with bad merges after accidentally merging the same change multiple times). And at the scale of repositories in the open source world, git was massively faster than svn. But the speed doesn't scale, and if you've got a monorepo you almost certainly aren't doing real decentralised development, and the merge tracking in subversion was fixed (well... maybe; honestly I haven't been paying enough attention to be sure). So seriously SvnHub would actually be a better basis for the monorepo world. It's almost unfortunate that git took over so comprehensively that now companies want to use it for a different context where it really doesn't shine at all.
Weird. Shouldn't that be just a no-op?
But to give an approximate explanation: Subversion's model of how branches work is fundamentally different to git's model. In fact in a sense subversion doesn't have a concept of branches at all in its core data model. Subversion just gives you a directory tree with a linear history, and then makes it cheap to copy directories around. The recommended layout for a subversion repository looks something like this:
- /tags
- /tags/release-v1
- /tags/release-v2
- /branches
- /branches/my-feature-branch
- /trunk/...
Some references for you:
- The classic "svn book" about branching: https://svnbook.red-bean.com/en/1.7/svn.branchmerge.using.ht... - note the section at the bottom "The Key Concepts Behind Branching" which notes "First, Subversion has no internal concept of a branch—it knows only how to make copies. When you copy a directory, the resultant directory is only a “branch” because you attach that meaning to it."
- The subversion 1.5 release notes explanation of merge tracking: https://subversion.apache.org/docs/release-notes/1.5.html#me...
To try to place this in time, I was using subversion (and was pretty happy with it!) in 2007. By 2009 I was using git wherever I could. Of course, the widespread migration toward git throughout the open source ecosystem was spread over a lot more years than this. But hopefully that gives some context.
Seems (at least the original version you described of) subversion was in some/several/many ways genuinely inferior to git.
But yeah, having thought about it a bit more re-applying the same change again in any versioning system is of course not necessarily "a no-op" if they're not immediately following each other; if other changes that partially reverted the original ones have happened in between. Then re-applying the original changeset would of course re-do that part of the original changes.
I think this is happening because git submodules have a hard and confusing CLI even compared to the rest of git.
> would have been more than happy to use "SvnHub"
svnmerge.py caused almost as many trainwrecks as it avoided. I don’t think resolving merge conflicts can be made to work so long as the repo relies on being informed of all file moves and copies, because devs just didn’t do that consistently.
> But why do so many people want these monorepos?
Google-copying is part of it for sure. And I agree with your position - copying Google isn't a great reason on its own. Some more valid reasons: code which deploys together often wants to live together. Common dependencies can be easier to find (and use) if they're right there in the repo. Related, making cross-cutting changes can be easier when you can make an atomic change everywhere. Also, big repos often started out as small repos and then grew over time; the cost of a major change might outweigh the friction caused by keeping Git in place.
Consider also that we might not all be talking about the same thing. Some people (even here in this topic) consider Linux to be "large" and "a monorepo". Linux isn't notably big anymore, and it's not unusually challenging to host or to use locally. It's arguably a monorepo since it contains most of the code for the Linux kernel, but to me, "monorepo" implies needing special attention. So I probably wouldn't classify Linux as a monorepo for this discussion.
> I can't figure out why so many people are married to git but insist on leaning into its weaknesses
I'm sure there are many reasons. The common one I hear in my role boils down to, essentially, "Git is the de facto standard". That can be expressed in several ways: "it's harder to attract and retain engineers with an uncommon toolset"; "we want to focus on our core business, not on innovating in version control"; "all the other tools in our kit work with Git". (NB: I put those in scare quotes to distinguish that they aren't my or GitHub's position, they're things I've heard from others. They're not direct quotes, though.)
I talk with customers weekly who want to mash dozens of independent repos (Git or otherwise) together. If they aren't going to reap any of the benefits mentioned above or elsewhere in this topic, I strongly advise against it. At the end of the day, GitHub doesn't care if you have one giant monorepo or 1000 tiny ones; the product works pretty well for both. I suppose that's why I felt compelled to reply to your thread in particular -- yes, we're investing in monorepos, but no, it's not because we're trying to drive people to them.
> Consider also that we might not all be talking about the same thing.
I think this is 90% of the confusion/disagreement in this thread.
I've always thought that it's fine to use git however you want, but if you hit a bottleneck (it gets too big and becomes slow), then you split your repo along logical boundaries (usually a large library can be split out and versioned, for example).
Somewhere over the last 15 years, that's changed, and the zeitgeist is now "mash dozens of independent repos into one repo" no matter the situation. Everyone in this thread that's suggested monorepos aren't the way to go has been downvoted, which caught me by surprise.
> I suppose that's why I felt compelled to reply to your thread in particular -- yes, we're investing in monorepos, but no, it's not because we're trying to drive people to them.
I believe you're sincere, and perhaps I was a bit too cynical about GitHub's motivations. Sorry about that, this topic is just frustrating and GH is in a position to help make things better.
Do you think GitHub could offer some kind of practical guide about when to use monorepos and what their limitations are?
I think part of the problem is that git's docs and the git-scm book aren't going to prescribe a way to use the software, because it's intentionally extremely flexible. Git users appreciate this, but GitHub users might lack good guidance.
As another reply pointed out, this might also have its origins in the git-submodule porcelain being confusing and underutilized.
Most GitHub users have probably never used submodules, don't know when a git repo would start to slow down due to size, aren't sure how to split out part of a repo while preserving history, and probably haven't thought too much about internal dependency versioning.
> to me, "monorepo" implies needing special attention
I think you and I are actually in total agreement, but the vast majority of corporate GitHub users have no context about where git came from, what it's good at, what it's limitations are, and how to use it for more than trunk-based development.
The ideas that "Linux is a monorepo" or "monorepos are the only natural way to manage code for any project" or that git should be "fixed" to support centralization should be concerning to GH.
I suspect these people don't have "monorepos" in the way that you and I are talking about them. They probably just have mid-sized repos that haven't need to be split up yet.
Even if you can support those customers as they grow into monorepos without friction due to enormous technical efforts, we're failing to teach a generation of engineers how to think about their most fundamental tools.
I appreciate that GitHub is trying to use technology to smooth out these points of confusion and improve git to work in every scenario, but publishing customer education materials about how to make good decisions about source code management would also help a lot.
> I talk with customers weekly who want to mash dozens of independent repos (Git or otherwise) together. If they aren't going to reap any of the benefits mentioned above or elsewhere in this topic, I strongly advise against it.
It sounds like you're giving good advice to the teams that you talk to, but what can I show my team that's authoritative from GitHub that says "don't mash dozens of independent repos together and then blur the lines so you can't tell what's independent anymore"?
I thought this was obvious, but it's not, and I don't know how to get people to understand it.
This is a particularly bad problem when your independent repos are working fine, but then there's a company-wide initiative to go "monorepo", and it's obvious in advance that the resulting monorepo won't be usable without a lot of extra work.
Maybe I've just been unlucky, but every time I've had a monorepo experience, it's been exactly that approach. And, as I'm sure you can tell by this thread, I haven't had much luck in convincing other engineers that mashing all unrelated code into one repo is a silly thing to do.
Everything about git is orders of magnitude slower than the monorepo in use at Google. Git is not fast, and its slowness scales with the size of your repo.
The "tool" in this case is a feature being added to upstream git, which everyone gets for free. It seems reasonable to assume other git providers will also be adding support for it server-side where needed; I have to imagine Gitlab either already is or will be doing work to support monorepos as well, assuming there's enough demand for it. At worst, this seems like it could be trying to lock people into git generally rather than GitHub specifically, but more likely this is just work being done to support something that (potential) customers are already asking for.
Didn’t find anything interesting.
Elon Musk.
https://en.m.wikipedia.org/wiki/Monorepo
Then you can do your own opinion, I'm sharing mine.
In smaller scales, for example, they're fantastic for productivity, and my company is not looking back.
According to multi-repo advocates, it would obviously be split up into at least a dozen different repos, with dependency management tools to specify which version of zlib or ebpf to use, with a dedicated repo just for the build logic, of course.
But you can find a list on Wikipedia and make your own opinion : https://en.m.wikipedia.org/wiki/Monorepo