I feel terrible for anyone trying to run a company with open-source style independent repos. On a popular github project, you have MANY potential contributors that will tell you if a PR, or a release candidate break API compatibility, etc. There are thousands of hours in open source dedicated to fixing integration issues due to the (unavoidable) poly-repo situation.
Monorepos in companies are relatively simple. You need to dedicate some effort in your CI and CD infrastructure, but you'll win magnitudes by avoiding integration issues. Enough tooling is out there already to make it easy on you.
Monorepos' biggest problem in an org is the funding, as integration topics are often deprioritized by management, and "we spend 10k per year on monorepo engineering" for some reason is a tough sell for orgs, who seem to prefer to "spend 5k for each of the 5 teams so that they maintain their own CD ways and struggle integrating which incurrs another 20k that just is not explicitly labeled as such".
Developer team dynamics also play a role. I have observed the pattern now multiple times (N=3):
* Developers have a monolithic repo, that has accumulated a few odd corners over time. * The feeling builds up that this monolithic repo needs to be modularized. * It is split up into libraries (or microservices), this is kind of painful, but feels liberating at first (now finally John does not break my builds anymore) * Folks realize: John doesn't break my builds anymore, but now I need to wait for integration on the test system to learn if he broke my code, and sometimes I only learn it in production. * people start posting blog posts on monorepos
That pattern takes 2-3 years to play out, but I have seen it on every job I worked.
with that in mind, is monorepo is a universally good approach or is more dependent on good behavior of team members than polyrepo?
Suppose for your projects you have a utility library `lib_a`, in a polyrepo situation, your projects will use it in probably different versions, which means you have coordination effort necessary to get everyone on the latest release. The monorepo would enable the developers of `lib_a` to get feedback from the downstream test suites directly on whether the changes they perform are breaking user code, so they can up front introduce their changes less intrusive. They can however also roll out security-relevant changes much more easily. The monorepo will make the projects more homogeneous, which facilitates integration and operations (there are exceptions of course).
Monorepo people ignore the learned lessons of those who came before us, and are trying to drag their teams back into a simpler time that, while nice, does not exist anymore. If you use any dependencies at all, you don't live in a monorepo world, and lying to yourself and your coworkers will only leave you confused and angry that your expectations are constantly not being met.
The solution isn't to split every single component into its own repo, but pretending like that's what anyone rational is proposing is not working with the best form of the argument. It's not always completely clear how to split up a growing codebase, but to claim that it's not usually worth splitting up is Wrong.
It's a frequent problem to conflate organization/modularization with lifecycle/version management.
You can have a well-organized codebase just as easily in a monorepo.
That's a separate question from management the lifecycle of the code. (What is release and when? What tests are run? What process approves a change?)
But people seem to forget that it wasn't that long ago that git didn't exist, making multiple repos was a pain in the butt. Managing multiple repos locally was hell. Monorepos were the norm.
Then as the state of version control ramped up, and making repos became easy, and having so much code in one repo had performance issues (overnight CVS/SourceSafe/SVN pull on your first day at work anyone? Branches that take hours to create?), people started making repos per project. The micro-service fad made that a no-brainer.
Now, for companies like Facebook and Google, or really any company that wrote code before the modern days and has a non-trivial amount of it, switching was not exactly a simple matter. So they just poured their energy into making the monorepo work. They're not the only ones to do it either (though not everyone has to do it at Google, Facebook or Microsoft scale, obviously, so its a bit easier for most). And so it works. And then people forget how to make distributed repos work and claim things like "omg I have to make 1 PR per repo when making breaking changes!", as if it was a big deal or it wasn't a solved problem.
(I’m also not sure I’d generally categorize tools work as as “dev ops,” though I can certainly see how they end up intertwined.)
You can have both tiny repositories which do a single thing and large repositories that consist of many projects. It's totally cool to have both, assuming your team can be trusted to make the appropriate choices as they create new projects.
> And then people forget how to make distributed repos work and claim things like "omg I have to make 1 PR per repo when making breaking changes!", as if it was a big deal or it wasn't a solved problem.
Is this a solved problem? I typically do make one PR per repo to resolve breaking changes, though it's certainly not a big deal. Still, if there's an easier way, I'd love to hear about it!
I don't mean that it's magical, just that it's not particularly sorcery. Instead of making a breaking change, add new method, deprecate old method. Update projects, then get rid of old deprecated method. Because they're distinct you can do this one by one so some project can reap the benefits without having to wait until all the problems are solved.
Some people in this thread act like its freagin impossible. Avoiding breaking changes in APIs or proper deprecation strategies is an art everyone developing software should know: sooner or later they'll have to contribute to an open source project or have to make a more complicated breaking change or SOMETHING and will have to deal with it. Even if they use a monorepo. And when it happens you don't want it to be the first time anyone deals with it.
I don't recall having such issues when I was working with Subversion and Perforce.
On the other hand, not everything was rosy in the 'good old days': MS Source Safe was (by far) the worst VCS experience that I have ever had.
Wouldn’t any project fall apart without devops work?
We have quite a few projects but only 4 major applications. Maybe it is that a few of our projects intertwine a bit so making spanning changes in separate repositories was a pain. Doing separate PRs, etc. Now changes are more atomic. Our entire infrastructure can be brought up in development with a single docker-compose file and all development apps are communicating with each other. I don't think we've had any issues that I can recall.
We are a reasonably small team though, so maybe that is part of it.
for example I want to use branch rev5 from project A and rev3 from project B
how I do that in a mono repo, I could not do it in HG, but sure about GIT
Not having it this way would be equivalent to having subrepo that refers to HEAD instead of a specific commit which is normally considered an big anti-pattern.
The best remedy is to not do branching like this in the first place, just try to stay on trunk all the time.
In SVN a branch is simply a convention. You copy (almost zero cost) things around into your branches directory
You do the usual 3-way merge thing to push your changes upstream, or pull upstream changes into your copy. As with git, the VCS tracks which revision of upstream your copy is up to date with, which is how it determines the base for the 3-way merge.
I should add that a huge amount was invested in tooling. We had an in-house IDE with debug tools that could step through serverside code. We had a highly optimized code search tool. We had modified a major version control system so it could handle our codebase. (Indeed we picked our version control system because we needed to fork it and the other major version control system was less amenable to our PRs.)
My current job we have a micro service architecture and lots of small, focused repos. Each repo is self-documented. Anyone can checkout and build anything. We don’t need obscene dev servers. We have not hugely invested in tools or workflow.
Client apps are unavoidably larger repos than the services apps.
Based on my personal experience, I think monorepos are nuts.
I get some perspective from the comments above. There is seemingly an army of engineers at Google that keeps the monorepo functioning. I was at a meeting about bazel and angular. I thought I'd ask how they do things at Google. To my surprise, the presenter said he is not at liberty to discuss how things work at Google. I guess it wasn't so surprising in hindsight. I mean what would I do with that information, right? It would be way too overkill for my tiny crud application.
company
/ProjectA
/.git
/ProjectB
/.git
company
/.git
/ProjectA
/ProjectB
The later forces me to be aware of the entire universe in that repo.
Maybe you are thinking of CVS? In SVN, creating branches has always been cheap in both space and time.
I've worked with monorepos, and I'd be loathe to recommend it as well; the combination of culture shift and tooling it takes to keep a monorepo system running makes most CD processes you see today look like child's play.
There is a lot of very good free software that supports most of the open source approach to CD these days; but very, very little freely available monorepo tooling. Just check out https://github.com/korfuri/awesome-monorepo - it's a quick read. I haven't found many other notably superior compilations. Compared with available OSS workflows and tooling, it's rather sparse, filled with bespoke approaches everywhere.
- https://github.com/facebookexperimental/mononoke - I hear this is a real thing and not a science fair project
- https://github.com/bors-ng/bors-ng - Needed in a monorepo to handle high arrival rate of commits / merges
I understand at google scale you'd need lots of tooling but why at a smaller scslr of merging a dozen small repos?
With all that integration, your single CI toolchain is front and center since everyone's success or failure is tied to it. While projects like bazel exist, how many developers do you know work with bazel every day? I no nobody who does. And most want documented IDE support and ease of use, not some optimal CI workflow. I've found gradle to be OK, but even that kind of pushes everyone toward using Jetbrains tooling. In the end, almost real monorepos have significant custom CI tooling that wires together different toolchains, and, they may have to maintain custom tooling for use in developer machines. And that custom tooling can get expensive to maintain as the project scales up.
My personal opinion: very few companies will hit a point where sheer volume of code or code changes makes a monorepo unwieldy. Code volume is a Google-problem. But every company will have problems with Github/Gitlab/whatever tooling with multiple repos; coordinating merges/deploys across multiple projects, managing issues, context switches between them, etc. And every company will also have problems with CI/CD in a monorepo.
Point being... there are problems with both, and there are benefits to both. I don't think one is right or wrong. I personally feel that solving the problems inherent to monorepos, at average scale, is easier than solving the problems inherent to distributed repos. The monorepo problems are generally internal technical, whereas the distributed repo problems are generally people-related and tooling outside of your control.
It stuck with me, and is applicable to so many things. Including, maybe, this?
I just looked it up, Facebook has 2.2 billion users monthly. That's almost a third of the entire planet.
Shit that makes sense for them won't make sense for 99% of everyone else.
Often the focus is extremely weird. When people noticed that WhatsApp only employed something like 45 engineers, then most assumed that it was because they used Erlang and FreeBSD. The thought that maybe their success was do to hiring the very best engineers and paying accordingly is less attractive.
Monorepos is just a another item to the heap of things that may be a good idea, but it depends.
Easy to use, cutting edge updates.
Some mad genius in a company will write a fuck-ton of helper classes and utilities that take the heavy lifting out of everything remotely hard, to the point where you almost never need to touch a third-party API for a CMS, email send service, or cloud-hosting provider. Instead of supplying these as private NuGet packages to be installed into an application, they sit in solutions in their entirety, in case they are needed. That application then goes to a new developer team, and they have zero idea why there are millions of lines of code and dozens of projects for a basic website that doesn't really seem to do anything.
It's a nice idea, but it has resulted in some very tightly coupled applications. I remember one time where a new developer changed some code in one of the utilities that handled multi-language support, and for some reason our logs reported that the emails were broke.
Linux kernel is a monorepo.
Imagine if we combined KDE, Gnome, Linux Kernel, ZFS etc all in the one monorepo.
Otherwise, it's an optimization; see "premature optimization" for cautions.
All the ways of splitting code up and deploying multiple git repos for one project seem terrible.
A monorepo in Perforce!
The entire programming world revolves around libraries and yet when it comes to our own code we are afraid of them ? Strange.
If it's one project, it's not a monorepo. It's a repo.
This seems difficult in git.
There are "submodules" and "subtrees" but none seemed particularly great and as far as I could tell each came with a bunch of caveats.
I'll admit my Git skills aren't great, but I've used a variety of source control and tried to suss out the best way to deal with a small team.
We ended up using "git subrepo" which is an add on thing I don't love, but it works.
part of the motivation is "common" and "project 2" are to be open sourced, but "project 1" which also uses "common" isn't.
Of course they are, git isn't the tool for this. You don't want multiple repos for a single project, you want one per project (this is not a monorepo). If there are things like code common to multiple projects then they are their own project with their own repo and release schedule, releases go into some sort of package manager (even if it's just the file system) and the projects depending on that common code update as they go.
I haven't lost anything, I've gained the ability to make breaking changes because I don't have to update everything that breaks all at once. I don't have to do it at all because that's the job of the team responsible.
With a monorepo what happens when their are 17 projects using the common code and I'm not familiar with 16 of them? Do I have to dive into the code of all 16 and fix them?
You can send your pull request to the affected team leads, and request that they approve it, once they make changes on their end.
I mean, the alternative is that you have 17 different projects, each using one of five different versions of the common code. Heaven forbid one of them makes an incorrect assumption about another. Getting 17 different teams to dance together around a breaking change is always going to be hard.
This is an issue that needs to be managed, from the systems I've seen it tends to be managed poorly, that's in both monoish repos and multi-repo setups as well as everyone using third party packages. I don't think committing everything to trunk is a good way to resolve it though, they only upside to this approach is that it might force you to resolve it.
What I have to deal with much more frequently is the opposite problem, we have an urgent update that will break several things but has to be deployed for one dependent binary ASAP and fixing the rest of the universe first is not an option.
Worst case it might create some security issues, something that should be a breaking change getting kludged into a new breaking change but still being broken.
The idea that clients can run on the old library forever is a nightmare, especially for security-relevant changes. When I see a binary built yesterday I want it to contain yesterday’s code, not a copy of libpng 0.1 from 1996.
That is one viable workflow: Make a change to the common code and publish it as a new package version while allowing all existing code to continue to use the old package. Then, migrate other projects to the newer version of the dependency one by one.
Allowing multiple versions of the same code to exist in production at once adds complexity. It's a trade-off.
Also, if you're doing this with code that is ultimately webpacked to run in a web browser and you don't pay attention to the full tree of dependencies you're working with, there's a chance you end up loading two versions of the same library into a single web page, increasing the page weight and possibly causing incompatibilities in event handling.
Google prefers to simply have one master version of the entire company at a time.
I've spent a lot of time wondering which solution is the best and I'm still not sure.
You probably should have a way to visualize bundle size increases in PRs easily, so that this becomes obvious. Alternatively, some package managers like Yarn let you flatten the dependency tree, forcing you to pick one version of everything. Even with a monorepo, since you'll likely be using 3rd party dependencies, it's always an interesting exercise because of how hard NPM makes this: getting to a point where you only have 1 version of every 3rd party package can be very, very hard as some combinations of libs are mutually exclusive.
I don't think there's a universal answer to your question.
It should be hard to make breaking changes in common code. Even 'trivial' breaking changes seem to have a way of breaking things even when they shouldn't. If you need to make a breaking change to common code, the proper way to do it is add the new functionality separately, deprecate the old functionality (i.e. with javadoc so it gets called out explicitly by the IDE), and incorporate it 1-by-1 into consumers until none are using the deprecated version anymore.
You should apply care when making breaking changes. Having it be hard is a separate issue - I'd say distractions from multi repo tooling would introduce more risks overall. Having a unified CI system in a monorepo is really nice.
Facebook patched Android Dalvik to increase the "max methods per app" limit.
Don't know if there's a similar iOs story.
They're whole schtick was "we're Facebook and we have unique scaling needs that no one else does," which makes absolutely no sense from the perspective of one user's content being rendered on their phone.
Plus the presenter was pretty smug, like all of this was good, when he wasn't convincing anyone that it was even necessary.
Found the slides: https://www.columbia.edu/~ng2573/zuggybuggy_is_2scale4ios.pd...
Also, why isn't such tooling available as open source? I'm trying to do my bit, but we could do with more effort being put into this, somehow.
I advice Google to replace the person in their internal IT who came up with that idea.
That's a huge understatement. They haven't just slapped a few scripts on top of git/svn, they've created their own proprietary scm to manage all of this. They've thrown more at this beast than most companies will throw at their actual product.
I'm also not convinced they haven't reinvented individual repositories inside this monorepo, it sounds like you can create "branches" of just your code and share them with other people with committing to the trunk, this is essentially an individual repository that will be auto deployed when you merge to master.
Now let's say I break a project sharing this code and because I'm not an expert in all 2 billion LoC and 3000 projects google is running I need to enlist some help in fixing what I broke. Presumably there is a way for the developers on that downstream project to pull in my change set? That's a shared branch.
Now assuming I can get all of these planets aligned correctly I'm going to need to take this set of changes and put it into the master version aren't I? That's merging my branch into trunk.
You said the first thing doesn't exist? Do you not have local changes or are these changes not shared with the test/build server? Having a set of patches, code changes, whatever sounds like a branch to me, are you being too literal with the word branch?
For the second part what doesn't exist? Do you not make changes that breaks other peoples code? Do you not get them to help fix it? Can you not share your work in progress changes with others? Can you goes collaborate on changes at all?
[Everything I'm about to explain is for the average user's workflow, like others have mentioned, "real" branches do exist, but most engineers will never use them, and my current workflow works differently than what I'm explaining, but I used to do it this way.]
Piper generally speaking doesn't have the concept of commits or "sets of patches". You have clients. A client trunk@time + some local changes. You could maybe call this a branch, but you can't stack multiple commits[1], so its a branch of length exactly one. It can only be merged back into trunk. Then you delete the client and start a new one. You can patch changes from one client into another, but this isn't generally done or super useful because again, you can't stack changes.
A given client has an owner, and the owner has write access. Everyone else has read access.
So to answer your questions:
>Do you not have local changes or are these changes not shared with the test/build server?
There are local (sort of) changes. And you can test/build them, but they lack many of the concept one would expect of a branch, so I'm not sure that's a good name for them.
>Do you not make changes that breaks other peoples code?
Sure you do. But you're responsible for fixing it (as I said elsewhere).
>Do you not get them to help fix it?
Yeah, but normally this is done by having them review the change, or talking in person. There's nothing like multiple commits by multiple people which are then squashed and merged.
>Can you not share your work in progress changes with others?
Sure, but they can't edit them.
>Can you goes collaborate on changes at all?
Kind of, but not with multiple authors.
[1]: There's a hack that allows chained CLs, but its a hack, a leaky abstraction, and still doesn't provide multiple authors squashing and merging.