Hacker News Reader

Comment by rzimmerman | Hacker News Reader

I've worked a bit with "the big monorepo" (though nothing like Google scale) and my impression is that a lot of the benefits fall apart if you don't have people working to maintain the build environment, automated tests, developer environments. When it takes hours or more to run tests and produce a build it can really slow the team down. The ability to force updates on dependencies in a single big commit can be really worthwhile as long as you're willing to spend the time to build the tools and do the maintenance.

View on HN

> a lot of the benefits fall apart if you don't have people working to maintain the build environment, automated tests, developer environments.

The thing is, a monorepo makes clear what multiple repos can obscure: if you have a software system, then you need all those things anyway. Multiple repos can hide the cost, but they don't eliminate it — in my experience, they multiply it. E.g. if someone makes a breaking change in a monorepo, he breaks the tests and he fixes them before his changes can be merged in; but with multiple repos someone can commit a breaking change and a dependent repo won't know about about it until days, weeks or even months later; the person responsible for fixing it has no idea what really changed or why; and he has a broken build until he does figure it out.

> The ability to force updates on dependencies in a single big commit can be really worthwhile as long as you're willing to spend the time to build the tools and do the maintenance.

It's like maintaining your car: you can spend $20 to replace your oil every 3,000-5,000 miles, or you can spend thousands every 10,000 miles. Up to you.

> but with multiple repos someone can commit a breaking change and a dependent repo won't know about about it until days, weeks or even months later; the person responsible for fixing it has no idea what really changed or why; and he has a broken build until he does figure it out.

Isn't that the whole point of versioning? If there is a breaking change only the repo's using the latest version will see it. At which point they can submit a bug report before any of that code enters production.

I still agree that avoiding all of that is preferred.

That would be great if people always used versioning correctly. I deal frequently with breaks caused by simple things like renaming, removing, or re-ordering protobuf fields in one repo, and not updating code in another. At least in a monorepo you can catch those sorts of errors.

On the other hand, some of the code you have to update may be for products that have been mothballed. That might end up being a waste of time if the project is never revived.

Or the code might be safety-critical and updating it is introducing unnecessary changes (and hence, unnecessary danger).

Some tests may include hardware integration. For example, aircraft software may need some number of flight hours after significant changes. That's probably not going to be a part of the CI suite, and the changes will introduce a greater manual testing burden.

That's why you delete dead code at head (keeping archive of history) , and you have dev branches and release branches

> a lot of the benefits fall apart if you don't have people working to maintain the build environment, automated tests, developer environments.

This is absolutely true. Without a very strong culture around testing and deep, continuous investment in infrastructure and tools, a monolithic repo would quickly devolve to being a giant midden heap of code.

How is that different on multi-repos ? The total amount of code is the same. You also need to keep build environments(not one but several), dependency management is much harder,etc.

I would say that any large project needs a "very strong culture around testing and deep, continuous investment in infrastructure and tools"

In my experience with small teams, maybe one person will take testing & automation seriously. If they've got a couple small codebases to worry about, it's possible for them to manage. One giant codebase would be too much work.

Obviously not the way it should be, etc, but reality bites.

So you've got a few small repos with good build infra,and many small repos that don't. Your overall system is still a sibling ship, and you have no economy of scale force multiplying your few people who take infra seriously

There are dedicated and volunteer teams that work on the internal tools and a strong testing and accountability culture. Everything is built from source.

"When it takes hours or more to run tests and produce a build it can really slow the team down. "

I don't think those are necessary artifacts of a monorepo. A huge chunk of code does not mean that every component there is connected.

In Google it is somewhat connected, actually. Sometimes engineers have to run ALL tests over the repo. And it takes hours (better be scheduled overnight!). Although they might have fixed that already.

Kinda. Depends on who you ask. :)

I still cannot wrap my head around this. What merit does forcing internal code to update a dependancy have? Having dependencies packaged like NuGet, and using an internal repository, you can always remove a particular version of a dependency if you must, and a build will fail if it cannot automatically use the latest minor version.

It means that I, as a dependency author, can make a breaking change and push not only the updates to my library, but also the updates to all my dependents in a single commit. Using a dependency packaging system like NuGet is a workaround for me not knowing who all my clients are. In a monorepo, I can have perfect knowledge.

  push not only the updates to my library, but also the
  updates to all my dependents in a single commit.

And what's more, you can run all their automated tests and address any problems your changes introduce. Which presumably means regular forced updates aren't the recipe for instability they would otherwise be.

> Using a dependency packaging system like NuGet is a workaround for me not knowing who all my clients are.

Between published manifests and search indexed source code, you would know who all your clients are, even without a monorepo.

take debian. you write a new version. push the new package. everyone else's build will fail until they update.

its as bad as you describe, but more organized :)

Should you know who all your clients are? That seems to be a workaround for not having an adequate testing framework or a contract stating what your code will do. Do open source frameworks know everyone's code that is using it?

Plus you do have perfect knowledge if your CI tools builds everything on your commit. Which it will do anyway if you have a monorepo.

> That seems to be a workaround for not having an adequate testing framework or a contract

Sometimes you get that contract wrong, and you don't realize it until you've shipped it. If you want to evolve an API by introducing a breaking change, you can do it at Google. You just have to migrate the clients with the API change. It means your APIs can both improve and stay lean, without needing to carry around legacy cruft, which is great.

It's not really an option in open source.

That's a good point about not carrying cruft forward. I've often seen on projects where there's crazy methods like sendEmailOn3rdBirthdayAfterFullMoon as someone somewhere needed it for a single project but who knows if it is still in use.

You will not have perfect knowledge. Even at Google they don't truely have a mono repo, as there is several other projects that live outside the monorepo. A build system will also detect changes and run test, produce a build, and notify maintainers, when you update a dependency. Monorepo is not needed for this.

"I still cannot wrap my head around this. What merit does forcing internal code to update a dependancy have?"

It allows a push model instead of a pull one.

You can know everything you will break ahead of time. More generally, you can know the impact of a change (performance, etc) before you make it.

Is it possible to do it without a mono-repo? Theoretically, yes, through some large federated something or other. These don't really exist ;)

Basically, it lets you have a library team that moves the world forward without causing pain (at least, for those reasons :P).

The latrge federated something or other doesn't exist because Google chose to build monorepo tools instead, and approximately no one else's code base is really big enough to matter. Why didn't/couldn't Google build the federated something or other?

The short answer is likely "It wasn't worth the cost/complexity/etc of that approach".

It's an automated build system, it already exists. I think it's more likely that Googlers decided that they had the discipline for a mono repo - time will tell if that hold true.

Not theoretically, with [OpenGrok](https://opengrok.github.io/OpenGrok/) you can register all of your git repositories and then search for the dependency declaration of your library across the entire company's source code, and then with that list you can use a simple bash script to autogenerate the git diffs to switch to the appropriate security patch for whichever minor version of the library they're using.

You can also use this exact same approach to create tasks on teams to upgrade their library if you've EOLed the version they're using. You can be as narrow (requiring everyone on the latest version) or as broad (having a band of versions allowed, in case the change is significant enough to require updates to how it is used, or if there's a major component of the system using an older version for performance-regression reasons you're still addressing on the improved library) as you want.

This actually simplifies your build system, as you can reuse the same standard open source tools that come along with your language of choice, which often assume they're running at the root of the project and can do what they want to the surrounding directory structure, and build-n-test cycles take mere minutes instead of hours, being isolated to only the component you're working with (and in the case of a library update across project, parallelized due to this isolation across build servers, so still much faster than the monorepo approach).

This also makes the codebase healthier, in general. Libraries and services need to be designed to actually have a clear purpose and interface; they can't cheat by grabbing source files from each other while breaking the expected interface, so refactoring their internals has a near zero probability of causing downstream issues. Services can be rewritten by standing up a parallel service, shadowing traffic, and validating equal responses between them. You can also fork and repurpose service code for another project and know exactly what code is required by it immediately.

These things are possible in monorepos, but they require a very high degree of diligence, and even moreso in a mixed-language environment.

In general, breaking everything into a separate repository is madness (see the modern Node.js ecosystem), so new services should start out as "mini-monorepos" where everything new they need that isn't already an available library should be contained within the service. Once a few services seem to be doing the same sort of thing, then you look at the commonalities between their implementations, learn from past mistakes (that we all make), and make the library to share between them, and allow future services to not have to reimplement that.

(This also has the advantage of making it easier to open source said library, since it would contain zero references to proprietary services, and requires no cleaning of the git history, and to get services to use your library, you already need to document it clearly.)

Frankly, the world Google is in with their monorepo is an amazing feat of engineering built upon the initial tools they had available ~15 years ago, but it really isn't an ideal situation, and there are better, more flexible ways of doing things now. I'd be sad if our industry hadn't improved this over such a period of time.

This actually touches upon a subject I hold dear. For .net development, I use SimpleInjector, not because it is the best and most flexible DI container, but because it is deliberately designed to make it impossible to do things that are wrong following best practice.

Sometimes it is annoying that it lacks the flexibility, but it always means we just have to do the refactoring now, rather than let the code rot. A Lead Dev should always keep this in mind, choose tools that make the right way the easiest and possibly only way forward. A mono repo is an example of a tool that gives superb knowledge and flexibility, but it requires some serious discipline to avoid a rapidly growing complexity.

You're not the first to make this claim, but it seems dubious.

It means the library authors must be knowledge expert on all projects using the library. How can you make changes to all the client code without knowing the code base well? Make change willy-nilly on codebase you are not familiar with on the basis that it uses your library?

That sounds like a recipe for disaster.

Or you need to get consensus and code review for each and everyone of those projects and have all tests pass for all projects. This look like slow-as-molasse progress. Submitting any changes would always be an extremely big endeavour.

The alternative is to have submit not affect other projects. You can do this in a monorepo by mandating that other projects not sync to head but to a given release tag of your library and only move to a more recent tag on their own schedule, after they've tested the upgrade. IOW, pretty much how a packaged release would work. Pretty much how one would have sync'ing to separate repos.

I fail to see what are the benefits or monorepo. The downside is either anyone can touch anything without being and expert and having an extremely long and complex submission process. (And woe to you if someone submitted something in the meantime... you have to redo your testing and fixing again?!?)

The world at large has lived with separate repos and explicit releases for ages. That's how third-party library are handled in every single software team I've ever seen.

"You're not the first to make this claim, but it seems dubious."

"It means the library authors must be knowledge expert on all projects using the library."

They must know enough to design good APIs or else you will fail anyway. If your apis require expert knowledge of the system to use correctly, you've probably done it wrong.

" How can you make changes to all the client code without knowing the code base well?"

Good API and good refactoring tools

" Make change willy-nilly on codebase you are not familiar with on the basis that it uses your library?"

With good tests and testing tools, i fail to see the issue.

"Or you need to get consensus and code review for each and everyone of those projects and have all tests pass for all projects. This look like slow-as-molasse progress. Submitting any changes would always be an extremely big endeavour. "

Nope, there are tools to manage the changelist process for large refactoring tools. My experience says velocity is not an issue in terms of things moving too slow, pretty much ever.

"The alternative is to have submit not affect other projects. You can do this in a monorepo by mandating that other projects not sync to head but to a given release tag of your library and only move to a more recent tag on their own schedule, after they've tested the upgrade. IOW, pretty much how a packaged release would work. Pretty much how one would have sync'ing to separate repos. " Try this sometime, and see what the lag time to upgrade in a large enough set of teams and environment is. It's "forever until someone breaks you".

"I fail to see what are the benefits or monorepo." I think they've been covered quite well. You just think they don't happen in practice, despite people with experience saying they do. That's okay. you don't have to believe them, but it's not clear what anyone could do to convince you:)

" The downside is either anyone can touch anything without being and expert and having an extremely long and complex submission process."

The first is an explicit feature. If everyone needs to be an expert to touch most of your codebase, that sounds like a bad codebase to me. Your latter is simply not correct. The rate of change is many times faster than any separate repo process i've seen.

" (And woe to you if someone submitted something in the meantime... you have to redo your testing and fixing again?!?) "

If it's broken, it's broken, whether it's packaged or not. The only thing you are talking about is whether you can tell it's broken before you get to the next packaged release.

"The world at large has lived with separate repos and explicit releases for ages. That's how third-party library are handled in every single software team I've ever seen. "

Yes, and honestly, it's been a complete and total disaster everywhere i've seen it, with tons of teams with forked, old, buggy versions, the inability to reuse code (or bugs) because of subtle third party dependency incompatibilities, etc.

I'm not sure why you claim it's a panacea. Neither is, but not for the reasons you've given.

The facts on the ground suggest that Google does not have trouble delivering high-quality software, so your arguments aren't very convincing. To me the biggest issue is that their customer-facing software is often not what people want, but that isn't really a development process issue.

To know if it is breaking you do not need the mono repo, but access to the build system, mono repo or not, access to the source alone if not enough. It will detect an update to a dependency and start a new build, run all test, and produce an artifact if it's successful.

Agreed. It's a huge dependency of monorepo that is often not spelled out directly. Probably the reason is that the people who successfully implemented monorepos already live so deeply in that continuous integration world, maybe even continuous deployment, so that they don't even know anymore that a huge part of the world has not moved there yet (despite trying for a decade).

Reply on news.ycombinator.com