On Monolithic Repositories

I've worked with monolithic and project-based repositories, and in practice I've found the problems with monolithic repos are less than the problems with project-based repos, and the benefits of monolithic repos are more than the benefits of project-based repos. Certainly there can be issues at a very large organisation with very many extremely large projects—but most of us don't work at those organisations with that many projects that large.

I think that having one large repo helps identify cross-project dependency breakage faster, e.g. on a small team without fully-automated integration tests by increasing the likelihood that the person or team who broke the integration notices rather than the person or team who maintains the affected components.

There's also the issue, as jacques_chester notes, of shared components, some of which are far too small too small the be their own projects and which don't necessarily make sense thrown into a pile with other projects.

Project-specific repos make a lot of sense from an organised, a-place-for-everything-and-everything-in-its-place perspective, but real life is often quite messy and mutable, and the proper organisation for a project can change frequently (as the article notes); there's no sense chiselling it into stone.

I'm very much a fan of monolithic repositories, because I have had horrible experiences working with project-based repos. I've been around long enough to recall discussions of why people should stop using CVS, and invariably one of the bullet points on that list is "CVS lacks atomic commits." Projects that use multiple repositories invariably fall into the trap of having non-atomic commits, and people who advocate multiple repositories have likely never had the fun task of trying to do archaeology on those repositories where the sudden non-atomicity is suddenly painful.

When I've brought this up before, people occasionally mention submodule or subrepositories, and those are equally broken. It makes big assumptions about how you're going to be organizing repositories (i.e., a strict tree), and if your design is not in that organization scheme, you're up a creek. For practical development, the subrepo tree effectively becomes one monorepo anyways: touch the innermost subrepo, and now you need to add a commit to all the outer repos to update the newer version of the innermost subrepo.

A saner way to handle repos is to recognize that it's not necessary for people to have the full history of everything in the repository stored on their local machine most of the time. This is something that SVN does better--you can checkout subdirectories of an SVN repo, but the commits are still atomic across the entire repository.

I'm sitting here running git gc and repack on about 50 repos right now, of varying sizes. We just actually combined two of our larger repos into one for productivity reasons, so this article resonates with me a little on that front.

I spend a lot more time on the build and administration side of things than the code side, and I personally prefer more smaller repos. Builds are faster and less error prone, less disk space is used overall (regardless of cloning scheme - I have used them all), and I do believe the separation and inherent difficulty aids quality at the expense of productivity. I'm about the only person in the company who does, though, and that tells me this discussion depends more on how you personally interact with source control than any abstract 'monolithic vs. not' ideal. Or that I'm crazy, but I refuse to accept that.

Small repositories are great for open source because they encourage code sharing. When your project consists of many pieces all owned by different, often volunteer, teams, then having one big repository is a barrier to that.

For example, Servo, the project I work on, consists of around 150 small repositories (in contrast to every other browser engine). Lots of them are maintained by the Servo project, but lots of them aren't. They're separate projects of which Servo is only one user of many. The fact that we can take advantage of the fantastic work of the Rust community by simply adding a couple of lines to our Cargo files has been invaluable in helping our small team get the browser off the ground. A culture of monolithic repositories would discourage code sharing, leading to more wheel reinvention and less code collaboration overall.

I think the model works well for Google and Facebook because they're big centralized companies with thousands of engineers under one management and reporting structure. But that's a far cry from the aggressive, fine-grained code reuse culture that the Ruby, JS, and Rust (for example) have fostered.

I've worked both at a large company (Facebook) that used the mono repo approach as well as a large company that uses the per project repo approach (Uber) and I have to say I'm personally a VERY big fan of the project based repo approach but every company is different. So is every team. If you're a small company with primarily one service and primarily in one programming language, the mono-repo way seems to be the best approach. On the other hand, if you're a company that has embraced a service oriented architecture, the per project repo approach is likely the way to go. Especially if your company is OK with services being written in a variety of different languages and so long as it is as easy to use open source code as it is to use third party code written within your org. It also goes a long way in supporting local (ie., laptop) development. Otherwise, the entire codebase would be too big to fit in RAM.

Disclaimer, these opinions do not necessarily represent the opinions of my employer.

Another way to look at this is that repos quickly ossify into unplanned Conway boundaries.

Where possible, the goal is to decouple software components by design, not by backpressure from the toolchain.

I've seen the many-repo approach. It's particularly frustrating on distributed systems when a shared component migrates from repo to repo like a sad ronin, sometimes alighting in some of them more than once.

It all comes down to the scale factor.

The monolithic approach makes more sense for big companies that have a large portfolio of projects, an army of software engineers and the resources to develop their own tools around a SCM. This solves several problems like multiple commits per minute, resolving ever occurring merge conflicts, code search queries and code sharing. Someone might argue that it can slowdown the development process by having to load tons of data on a local machine but this issue can be solved by creating the correct custom tooling to download a sub node of the overall repo. At this scale level this is the only feasible solution to empower developer productiveness and to avoid headaches in keeping track of a long array of projects.

The project-based repositories solution has nice qualities as well: Service oriented projects, clearcut responsibilities and clear dependencies. This seems to be a good solution for a small to middle scale organization. Can design a well documented interface for every service and even expose some of those services to external clients when needed https://www.nginx.com/blog/microservices-at-netflix-architec... . Not that this service design solution is incompatible with a monolithic approach, is just that is easier to come up with a simpler answer when having silos of knowledge.

At the lower scale, where most startups reside. Either one seems to be a reasonable approach. Albeit a startup doesn't have the resources nor the time to spend on developing custom tools to manage a monolithic repo, it turns out they don't need to, because they have a small amount of developers and ongoing commit traffic. A plain simple git repo works as well as a multi repo. This is a matter of taste and the organization the founders/early employees wish to create for their project.

Option C: Custom tooling built for the purpose of managing collections of repositories.

It appears the main argument for monolithic repositories is that it improves developer productivity by giving the developer access to the entire organisation's codebase.

What a terrible hack to provide something that could be better managed with other tools. No DCVS is an island.

Someone needs to do for subrepos what git did for branches. Working with subrepos is a nightmare.

I work on the Developer Infrastructure team at Airbnb — essentially our tools team — and have some experience with both sides of the coin. Airbnb's monolithic vs project-based repository organization is currently split along language lines on the backend: the Java folks prefer a single monorepo (and have one), and the Ruby engineers use project-based repos (and have many).

There are a lot of good points made about the benefits of monorepos, and at Airbnb we enjoy several of them. What hasn't been mentioned is the effort required to do them well: you need specialized build and dependency tools to ensure that you only run builds+tests for the single project that's being changed; engineers have to check out extremely large amounts of data to work on a single subdirectory, or else you need custom tooling to allow them to only check out portions while still contributing to the larger whole; if someone mistakenly breaks a piece of shared code and merges it to master, every project is now broken and engineering work may be stalled unless you have very good debugging tools and testing frameworks to quickly recover from and prevent these kinds of issues.

The upfront costs of doing monorepos well are high, and doing them poorly is in my experience a net productivity loss. For large companies with established business models (and I'd consider Facebook, Google, and Airbnb to be some degree of "large" and with "established business models," although Airbnb is clearly still much smaller and pre-IPO), the tradeoff of allocating some number of engineers to work on in-house tools for a much larger engineering team is usually worthwhile, and monolithic repos start to become an attractive option. I'd caution small companies or early-stage startups against monorepos, though: when you're a twenty-person team, that amount of tools work just isn't worth it. Use open-source tools, and spend the rest of your time shipping product.

TL;DR: Facebook and Google have optimized their workflows for their size; if you're not a Facebook or a Google, your mileage will probably vary.

Architecture is just dividing pain into different buckets.

Large repositories are painful, as each developer bears the pain of integration every time they have to make a change to the repository integrating with all of the other code.

Small, distributed repos are painful when change accrues in larger increments and must be resolved at release/integration time.

As a developer in one of the largest monolithic repos in the world, I feel the pain every day of that pattern.

As someone already said, it is not "many repositores vs one" but often "many repositories + custom tooling to merge, agregate, diff, checkout, etc. vs one".

We've tried both, had one large repo (2G in size, mostly due to historical mistakes of checking in large binaries). Now we have many little repos but we've spend I don't know how many man months writing custom scripts (based on gyp) to manage all these collection of repos.

So far I would say it is toss-up. I would say for completely separate products that do not share almost any code it might makes sense to have separate repos. But if you find yourself building a custom version of git porcelains on top of your multiple repos -- you have probably gone too far.

Interesting comment on that article by Siddharth Agarwal:

"(I work on source control at Facebook.)

Every problem you mentioned with monolithic repositories is a well-known problem with Git (though some of them do have workarounds, such as clone with --depth). None of them are issues in principle.

With Mercurial we're aiming to provide tooling that scales well while still maintaining DVCS workflows like local commits."

The most interesting takeaway from me is that Mercurial is planning to support "narrow clones" soon, i.e. sparse fetches (i.e. you only have to download some subtree of the whole repository). It would be great if Git followed suit at some point - the whole monolithic versus project-based repository debate would be a lot more interesting if it were just a matter of convention, as opposed to the former being associated with "ew, SVN/Perforce/[insert old fashioned feeling tool]".

I'd agree with this article's conclusions if the current-gen VCSes (like Git and Mercurial) actually provided the necessary features and tooling to effectively manage a monolithic repo. Comparing to SVN's ability to checkout portions of a repo instead of the whole repo doesn't do much good in systems that aren't designed to handle that sort of workflow.

Whether or not one should go monolithic or project-based depends on how tightly integrated those projects are. Folks like Google and Facebook and OpenBSD's dev team develop all their things under a single source tree because those things are pretty tightly integrated and designed to interoperate. Other folks develop each of their things in a separate repo because they expect those things to be independently useful (OpenBSD's subprojects do happen to be independently useful, but AFAIK the priority is generally "OpenBSD first, everyone else second").

Multi-repository environments can also be very manageable depending on the language, runtime, etc. Dividing things into do-one-thing-and-do-it-well chunks (perhaps gems in Ruby land, or crates in Rust land, or packages in Perl land, or whatever) goes a long way toward alleviating the typical non-atomicity of multi-repo setups. Of course, this tends to be easier for open-source projects than closed-source projects (though the closed-source camp can have some of this fun, too; using Ruby as an example, one can run a private gem server just by running the "gem server" command, or - if using Bundler - can even install dependencies directly from Git repos), but it's still a possibility.

Plus, the whole "well Google says it's more productive for them, so we should take it as a general rule" vibe doesn't really sit well with me.

I am definitely in the multiple repository camp, but with one caveat (below).

With the monolithic approach everybody is forced to be on the latest version (or to update to the latest version). There can be a lot of contention when trying to get things out - especially when things are moving fast. You need a lot of discipline to make it work and you also are forced to constantly update parts of the code you own. Also, we tend to forget about deployment and management of the build artifacts. I am willing to bet that bigCo's that do this have a separate system for managing the artifacts and performing the deployments.

In the presence of a decent build and deployment system the only way to go is multiple repos - a logical separation per component / service. The build system can easily track dependencies via components via metadata associated with each of them. You can easily orchestrate builds when something happens, it's easier to rebuild only what changes and you get granular control over build artifacts.

can someone talk about how you keep track of commit messages when committing to different subfolder/subprojects of a monolithic repository ?

We have 6 projects on bitbucket - and we would love to move to a monolithic model (we have already seen integration issues because of developer carelessness with deployments, that would go away)... but I'm just not sure on how we will do things like Slack/Hipchat integration, emails on commits, etc.

Were I work we have a monolithic subversion repo that contains :

- DBs ddl

- Selenium automatic tests

- A hierarchy with our Java code were we have our libraries, modules, final products and client personalization of our products.

Our build solution relay on Maven with a nexus server and Jenkins to autodeploy maven artifacts.

Pros :

- We can do a single commit across all to update many modules/libs/final product to fix a issue or implement something new. So tracking a issue fix or a new feature across all is more easy.

Contras :

- We can't work like in git, where we can create a branch to develop a new feature or fix a issue isolated from other changes.

- Enforces big commits.

Actually I managed to use git svn to map our subversion repository, but I need to create many small git local repositories to track every module/lib/product/personalization that work over. Also, I have a few scripts to update my git local repos (master branch) against the subversion repo. This allowed me to be more flexible when I work on a new feature or a potential dangerous change, as I can create local branches, do local commits (and do a commit squash of they before send to subversion), stash , etc.

I think that with this I grab the best of both ways of working. I can be more flexible and more productive, as I can change of branch of a project/module/lib with a few keystrokes, and do local code versioning. At same time. I only miss doing a commit across modules/libs/products when to resolve a issue I need to do change across many modules/libs. On this case, I try to use the same commit message or at least, put the issue number on the commit message, so make easy track it on fisheye.

We use Perforce (since before DVCSs came along) and follow the monolithic approach with 100s of projects. We use git and mercurial when interfacing to clients' repositories and for small, local temporary repos to structure work that shouldn't clutter our main repository.

The monolithic approach works so smoothly because Perforce makes it trivial to only check out the directories we're working on. It has huge advantages when we're working on interdependent projects.

Perforce does have some historical oddities (top level directories are called "depots" and are slightly different than normal directories), but the ability to branch, merge, and check out using normal filesystem concepts is a huge usability boon.

I was a proponent of project specific repositories before I actually had to work with large scale system that consistently handled development in this way. This has me taught the hard way that managing transitive SVN externals (project A refers project B as external which refers to project C as external; I've had a hierarchy of 5 levels) is an incredibly tedious affair (tagging to get a consistent, pegged state of the main project is now a multi hour process instead of a single mouse click).

That alone justifies a single repository.

TortoiseSVN has a feature of pegging externals at a specific release when tagging the main project, but this only works at one level.

I agree with the author... In the context of startups that are very much in "discovery" mode (still doing big changes to their product) breaking the repositories in subparts is just a source of confusion for the tech lead. I would also argue it is bad for devs because they then need to manage version-binding between different repos. Many repos approach is only justifiable if you have many teams with many tech leads each independently doing releases.

How do people do CI with monolithic repositories? Do they run ALL the tests or is there an easy way around that?

I feel like I'm in the minority here; even my dotfiles live in separate repositories, managed by vcsh (https://www.github.com/richih/vcsh).