Hacker News Reader

Comment by acomjean | Hacker News Reader

But what are the alternatives to the monorepo in git?

All the ways of splitting code up and deploying multiple git repos for one project seem terrible.

Fun fact. I asked Facebook why they built their monorepo on Mercurial instead of Git. They said there were scaling issues in Git that made it unusable for large repos and the Git maintainers would not work with them to fix these issues. However, they were able to work with Mercurial to make it capable of holding their entire company in one repo.

Someone from FB did a really cringe-inducing presentation a few years ago about how "X can't handle our scale" (I think the predicate was iOS, but they went into IDEs and SCM systems). They had to pull the video and slides because it was so bad.

XCode also had issues working with large repos. Perhaps they were talking about that?

IIRC their iPhone app had about 20000 classes for some insane reason, and the system didn't handle that well.

https://www.facebook.com/notes/facebook-engineering/under-th...

Facebook patched Android Dalvik to increase the "max methods per app" limit.

Don't know if there's a similar iOs story.

Responding late, so you might not see this. The thing that was so ridiculous to me was Facebook pretending that their app is somehow orders of magnitude more complicated than everyone else's.

They're whole schtick was "we're Facebook and we have unique scaling needs that no one else does," which makes absolutely no sense from the perspective of one user's content being rendered on their phone.

Plus the presenter was pretty smug, like all of this was good, when he wasn't convincing anyone that it was even necessary.

Found the slides: https://www.columbia.edu/~ng2573/zuggybuggy_is_2scale4ios.pd...

Here's the article (or one of them) https://code.fb.com/core-data/scaling-mercurial-at-facebook/

> multiple git repos for one project

If it's one project, it's not a monorepo. It's a repo.

We wanted to have a "common" subsystem that was common across projects. Being able to add and work on the common area and new projects at the same time was important. Pushing the common area back and being able to deploy to the older projects and test was important.

This seems difficult in git.

There are "submodules" and "subtrees" but none seemed particularly great and as far as I could tell each came with a bunch of caveats.

I'll admit my Git skills aren't great, but I've used a variety of source control and tried to suss out the best way to deal with a small team.

We ended up using "git subrepo" which is an add on thing I don't love, but it works.

part of the motivation is "common" and "project 2" are to be open sourced, but "project 1" which also uses "common" isn't.

> what are the alternatives to the monorepo in git?

A monorepo in Perforce!

I don't understand why people are against this. You can have per repo branches/tags, the history is clean and relevant, it's easy to triage breakages, easy for different apps to have different versions of code etc. Plus for CI/CD it's trivial to just have one Jenkins jobs per repo as well and simple Git commit triggers.

The entire programming world revolves around libraries and yet when it comes to our own code we are afraid of them ? Strange.

> All the ways of splitting code up and deploying multiple git repos for one project seem terrible.

Of course they are, git isn't the tool for this. You don't want multiple repos for a single project, you want one per project (this is not a monorepo). If there are things like code common to multiple projects then they are their own project with their own repo and release schedule, releases go into some sort of package manager (even if it's just the file system) and the projects depending on that common code update as they go.

See, this is where your argument broke down for me. Once you’ve decided there is some library of common code, and assuming you factor out that code into another repo, you’ve just lost your ability to easily make breaking changes to the common code, which is something trivially easy to do in a monorepo. Why would you want that? It seems to me that if you have multiple projects sharing a base of common code then a monorepo is clearly superior.

> you’ve just lost your ability to easily make breaking changes to the common code

I haven't lost anything, I've gained the ability to make breaking changes because I don't have to update everything that breaks all at once. I don't have to do it at all because that's the job of the team responsible.

With a monorepo what happens when their are 17 projects using the common code and I'm not familiar with 16 of them? Do I have to dive into the code of all 16 and fix them?

You can soft-deprecate the old code path, and communicate that warnings will turn into errors at some future date.

You can send your pull request to the affected team leads, and request that they approve it, once they make changes on their end.

I mean, the alternative is that you have 17 different projects, each using one of five different versions of the common code. Heaven forbid one of them makes an incorrect assumption about another. Getting 17 different teams to dance together around a breaking change is always going to be hard.

If the common code is a versioned package, then each of the 17 different projects could update their code to handle breaking changes in the common package independently and update the version dependency after thorough testing.

You can have versioned packages inside a mono-repository, too, though. /common_libs/foo_lib_v1.13/, /common_libs/foo_lib_v1.14/, etc.

By that point your creating micro-repositories in your mono-repository and getting the worst of both worlds.

No, you're getting the best of both worlds, because it's incredibly clear to the infrastructure maintainers what version everyone's on, whether or not the old version can be safely deprecated, who is responsible for deprecating it, etc.

Yes, that's what you do. And you commit the change when all the tests pass.

And that's insane, people don't scale like that. It's harder enough keeping your head around one large project let alone every project a company has that you might have to jump into at any point.

What exactly makes automated refactoring difficult?

You're arguing for the nonexistence of something that obviously exists. There are tens of thousands of engineers at Google working in this manner on one of the largest codebases ever assembled.

I don't have a horse in this race but if you have, say, a security issue and that needs to propagate downstream where does your responsibility end in that situation? Do you try to track down the dependencies and open issues in their trackers? Or maybe a more common problem is a change to a library that consumes an API that's changing so updating the library has a drop dead date.

As a data point, we put the dependency graphs in a database at build time. When we have an emergency and need to push a library update to thousands of repos, we make the change, then trigger builds for all the dependents. We don't auto deploy (too risky), but we use the data we have to start nagging the owner of all the repositories to tell them they have to deploy asap. Since all projects are very small, they build and deploy very, very quickly (a few minutes at most for the big ones).

> I don't have a horse in this race but if you have, say, a security issue and that needs to propagate downstream where does your responsibility end in that situation?

This is an issue that needs to be managed, from the systems I've seen it tends to be managed poorly, that's in both monoish repos and multi-repo setups as well as everyone using third party packages. I don't think committing everything to trunk is a good way to resolve it though, they only upside to this approach is that it might force you to resolve it.

What I have to deal with much more frequently is the opposite problem, we have an urgent update that will break several things but has to be deployed for one dependent binary ASAP and fixing the rest of the universe first is not an option.

Worst case it might create some security issues, something that should be a breaking change getting kludged into a new breaking change but still being broken.

At Google yes, you would be expected to fix everybody. This will also give you exposure to client use cases and form a good basis for arguing your change is safe, necessary, and correct.

The idea that clients can run on the old library forever is a nightmare, especially for security-relevant changes. When I see a binary built yesterday I want it to contain yesterday’s code, not a copy of libpng 0.1 from 1996.

Someone will need to update those 16 unfamiliar projects, whether it's you or those projects' owners. From what I hear, Google's process is that the developer making the breaking change has to either update the other projects, or coordinate with the project owners, before they can check in the breaking change. It helps that Google has standard build and test systems for all their projects in the one monorepo.

What you're proposing goes a step beyond multiple repos and into package versioning.

That is one viable workflow: Make a change to the common code and publish it as a new package version while allowing all existing code to continue to use the old package. Then, migrate other projects to the newer version of the dependency one by one.

Allowing multiple versions of the same code to exist in production at once adds complexity. It's a trade-off.

Also, if you're doing this with code that is ultimately webpacked to run in a web browser and you don't pay attention to the full tree of dependencies you're working with, there's a chance you end up loading two versions of the same library into a single web page, increasing the page weight and possibly causing incompatibilities in event handling.

Google prefers to simply have one master version of the entire company at a time.

I've spent a lot of time wondering which solution is the best and I'm still not sure.

> Also, if you're doing this with code that is ultimately webpacked to run in a web browser and you don't pay attention to the full tree of dependencies you're working with, there's a chance you end up loading two versions of the same library into a single web page, increasing the page weight and possibly causing incompatibilities in event handling.

You probably should have a way to visualize bundle size increases in PRs easily, so that this becomes obvious. Alternatively, some package managers like Yarn let you flatten the dependency tree, forcing you to pick one version of everything. Even with a monorepo, since you'll likely be using 3rd party dependencies, it's always an interesting exercise because of how hard NPM makes this: getting to a point where you only have 1 version of every 3rd party package can be very, very hard as some combinations of libs are mutually exclusive.

It almost certainly depends on the company. Consider General Electic's microwave oven firmware and hydroelectric control station software. Both might actually share some code. Maybe they both use GEEventLoop or GEHardwareButton or something. But there's no reason to be concerned about having different versions in production at once.

I don't think there's a universal answer to your question.

If you are making a breaking change to a public interface you maintain, wouldn't you want to know how that interface is being used first, before justifying such a major breaking change? Not just change it for the sake of your own libraries internal convenience and hope that users of the library adopt. Since you know how the api is supposed to be used and all its best practices, understanding how to change the parts of the 16 projects using it should be quite easy, you shouldn't have to dig into the domain knowledge for all those projects.

> you’ve just lost your ability to easily make breaking changes to the common code

It should be hard to make breaking changes in common code. Even 'trivial' breaking changes seem to have a way of breaking things even when they shouldn't. If you need to make a breaking change to common code, the proper way to do it is add the new functionality separately, deprecate the old functionality (i.e. with javadoc so it gets called out explicitly by the IDE), and incorporate it 1-by-1 into consumers until none are using the deprecated version anymore.

And you can do that in a monorepo. But realistically, there are plenty of trivial breaking changes (renaming Foo to FooX) that don't warrant that effort, and so usually don't get done outside of monorepos.

You should apply care when making breaking changes. Having it be hard is a separate issue - I'd say distractions from multi repo tooling would introduce more risks overall. Having a unified CI system in a monorepo is really nice.

Reply on news.ycombinator.com