Benefits of a Monorepo
pspdfkit.com
pspdfkit.com
The thing I don't see discussed often enough, in all of the "monorepos are good/bad" debates, is that they require a certain discipline. When one commit can span multiple components, it's easy to add or ignore bad coupling between those components. That contributes to the rebuild treadmill for everyone, and is bad design in a bunch of other ways as well. IDEs make it worse, because they obscure the boundaries between components, but everyone ends up using an IDE because their predecessors made it very difficult to follow the flow of control across components otherwise. So you end up with one big ball of mud. With separate repos this is discouraged, because violating the rules of good design causes immediate pain. With a monorepo the pain must come in the form of stronger reviews, to discourage bad coupling even though it's easy. Unfortunately, the very same companies that have embraced monorepos have also abandoned separation of concerns as a design principle. :(
ETA: please read bunderbunder's comment (https://news.ycombinator.com/item?id=19796960) as well. It's an insightful take on some of the same issues.
Development happens at HEAD is the important part.
Another thing you can do is have your CI only check out specific directories when running tests using something like git’s sparse-checkout.
A monorepo oriented build tool such as Bazel will make sure that you don't compile more than needed. This is extremely efficient. In fact, in my company with a few hundred projects we tried it and it was 10 times faster than our previous Maven based workflow. These tools also help you with dependency management so you keep your code organized. The killer feature is that you finally have true Continuous Integration because the build tool will figure out any dependencies of the code that you change, rebuilding it and testing to make sure that nothing is broken.
Not at all true in my experience. I use Buck, which is very similar. It constantly recompiles stuff it doesn't need to by any standard. I've seriously considered avoiding Mercurial bookmarks in favor of manual patch management to see if I can avoid triggering the cases where it gets stupid. Maybe it's a problem specific to Buck, but my conversations with peers at other companies suggest not (and only a tiny percentage of developers at any of these companies get a choice of which tool to use).
The other issue is that "more than needed" is subject to more than one interpretation. There's "more than needed" as code was actually written, and there's "more than needed" as code should have been written. Tooling can help with the first but not the second. If developers are sloppy about how their header files are structured, then a change to one that's widely used will make it "necessary" to recompile a lot of code - including code that didn't actually use the part that changed. Oddly enough, though, that "necessary" recompilation wouldn't occur if those same header files were split up. (BTW some languages make good design more difficult by requiring private implementation details in public header files, but that's a separate conversation.) Tooling can optimize what it can see, but what it sees might already be a tangle of dependencies added only because a monorepo made it easy.
To find out how/where the dependency was introduced, you can use bazel query 'somepath(your_target, some_other_file)'. Not sure if Buck have an equivalent, but bazel should work with Buck repo.
Sometimes it's unintentional, for example an include left in after it's no longer needed. Other times it's intentional but lazy. For example, I often see commits that pull in one dependency just to use one tiny little utility function, but that dependency pulls in ten others, which each pull in ten others, and before you know it any change in a hundred files forces yours to be recompiled even when that tiny utility function wasn't touched.
A responsible developer would refactor the header file to keep the number of spurious rebuilds down, but the world is full of irresponsible developers. With multiple repos, the public immutable part of an API and the private fluid part tend to find their way into separate files pretty quickly. With a monorepo they don't, which is kind of ironic because one of the purported benefits of a monorepo is that it's easy to reach into "someone else's code" and refactor it. Unfortunately, people just don't seem to use that ability for good often enough unless review culture forces them to.
In practice this is almost never because of "a monorepo" and more because people don't like being good about including what they use (see: any Titus Winters talk ever). If, for example you have :test_foo and :foo_lib, things are fine and dandy. When you modify foo_lib you'll run :test_foo and nothing else. But often people are lazy and you have :foobarbaz_lib which is built from three source files, foo, bar, and baz. Each of those has a test, but all the tests depend on :foobarbaz_lib, because that's all that exists. So when you change foo, you run :baz_test, even though in practice, baz might depend on foo, and not the reverse. You fix this by making :foo_lib, :bar_lib, and :baz_lib each with their associated :target_test and explicit dependencies, and then :foobarbaz_lib depends on all three libs (or doesn't exist at all, and everyone who needs :foo_lib uses :foo_lib directly.
So yes, absolutely, when/if you start (essentially) globbing your imports, bazel/buck/pants stop being awesome. So don't do that.
I think you mean in the practice you've seen. I've seen some practice too, and it seems much more prevalent in monorepos. Yeah, still a small sample, correlation does not equal causation, yadda yadda, but I still think it's because monorepos make it so damn easy to reach out and add a huge tree of dependencies without having to think about it. My whole point is that it's possible to do the right thing, but requires more discipline.
Keep in mind that the alternative is "I have to build (or pull a prebuilt version of) the entirety of every repo that I depend on". An entire repo is always going to be larger than a single target. Yes you're right that pulling in the entirely of {{some large tool}} because you want `tool.strings.util.mangle_name` is bad.
The alternatives are
1. (multi-repo) You stick mangle name in its own repo (think `left-pad`)
2. (monorepo) Depend on `tool` in its entirety
3. (either, though more common in multi-repo) Everyone re-implements mangle_name. If the way the name is mangled is important, and it changes in the future, you have major issues and incompatibilities. (think IPV4 -> IPV6 for an example where a "global" `parse_addr` function would have made everyone's lives easier). You can bet on the assumption that mangle_name will never change, but then you aren't doing software engineering[1]
All of these are the wrong solution
4. Core utilities like this get factored out into their own files. In a multi-repo environment, everyone depends on the entire core/util repo. In a monorepo environment, you depend on the granular files you need.
4 is the only correct solution, and note that in a monorepo environment it results in you depending on fewer things than in a multi-repo environment. Yes there are a lot of ways to do things wrong, But that's true no matter what tooling you use. When used "correctly", monorepos have the best outcome, and the bad outcomes aren't actually that bad (sometimes longer builds and binary bloat if you don't tree-shake). You always need discipline. Monorepos don't need more.
For the local edit-test cycle incremental builds are great, but any decent IDE will do them in the multirepo case just as in the monorepo case.
Doing a clean build for the final artifact that is going to be released is pretty standard practice, IME.
Incremental builds are great for quick day-to-day development, but in my experience they are invariably unsuitable for robust production builds. For me, those should use a git clone or equivalent into an empty directory and then a completely clean build of everything, always.
I wouldn't care if it was reliable. My experience is that it isn't.
> If you don't trust bazel, people have been using distcc+ccache since forever.
And it's been causing inconsistent builds all the while.
I think you're falling into the trap of "past solutions didn't work, therefore the problem is intractable", which is a special case of the "throwing the baby out with the bathwater" fallacy. Sometimes a new tool comes out that fixes the problems of past tools.
Where's the breakthrough then? What's the magic that allows bazel to succeed where dozens of similar efforts have failed? Sometimes a new tool does fix the problems of past tools, but that's almost always due to some new insight.
* Bazel's FAQ: https://bazel.build/faq.html
* A description on how Blaze works: https://google-engtools.blogspot.com/2011/08/build-in-cloud-...
Unfortunately the design doc for Blaze is internal and AFAIK not available publicly. I imagine they took a look at a number of the similar efforts and made decisions based on what those approaches did wrong.
If you think about it, a package manager workflow (like maven, nuget, etc) is also a form of incremental compilation. You compile your libraries only once and publish them in binary form in some sort of repository and then import them. The difference is that if you want to make a change in both a library and an application this becomes hard to do with packages, while very easy with a monorepo and Bazel.
The difference is that that's explicit and visible, rather than being "magically" figured out by the build system. And the compiled binaries are 100% reproducible everywhere since it's literally the same file from the repository everywhere. So it's much more reliable, understandable, and debuggable.
> The difference is that if you want to make a change in both a library and an application this becomes hard to do with packages, while very easy with a monorepo and Bazel.
Having a library and application so tightly coupled that changes have to be simultaneous is poor practice, and will likely get you into trouble when the time comes to deploy them. Better to enforce that you test old application + new library (or vice versa) up front, than hit it for the first time in production.
Yes you can post a PR that relaxes someone else’s control but they can block it.
Multirepo teams live with the wrong modularity boundaries longer than they should, because fixing the design ("violating the rules," when they're bad rules) causes immediate pain. In monorepo it's easier to do the refactor in an atomic commit.
It's all part of the sandboxing cycle. https://xkcd.com/2044/
I thought of having a mono repo at home because many of my projects are inter related but instead I spent some time with a library setup. For Common Lisp, I used a pattern in one of Zach’s blogs to have one central repo for all of my personal Common Lisp code that is configured as a root for Quicklisp. Any of my code can be quick loaded no matter where I am working. Really simple but it makes it possible to write small utilities/libraries and reuse them for whatever I am doing.
I sort of do the same for Python, writing a setup.py for each small project or library to install globally, including appropriate executable scripts. This also allows me to combine small things easily. I am not yet set up to do this yet for my Haskell projects but it is on my todo list.
If changes regularly involve multiple submodules, then I know that the boundaries are wrong and refactoring is needed.
I mean, if you have a single product split over many repositories then you will run into the issues mentioned by the author: A change will invariably involve multiple repositories. Again, to me this seems to be just an indication that you are, in fact, working on a single product.
My motto is: "What you release together, you should version-control together".
Maybe it's my old-timey way of releasing software, but because I do releases as RPMs, it bothers me less if you do things like have your base library, your backend services, and your frontend in the same repo. If they need to be able to be deployed independently, I just split them up as subpackages.
I guess most language package managers don't really have such a concept, so people are optimizing for that...
My experience is that testing the interactions of historic versions of different components in a mono-repo against each other is difficult. With a multi-repo set-up, one can checkout to whatever historic version one needs to test. That makes it trivially easy to set-up testing that allows one to test historic versions of components against each.
Of course, what in the end really decides about the strength of your concern separation is the shape of your CI, but alas, repository granuality is just much more visible then CI checks.
If you put frontend and backend into different repositories, you need to make two PRs for every new feature and -- for much worse -- you make integration testing much harder as you don't have atomic product merges to run automation against.
Don't laugh about this example, my corporation is doing stuff like this (backend only repo tightly coupled to frontend repo) all the time.
So maybe we can steelman the argument into "good vertical concern separation doesn't force you into doing multiple PR for single feature, as opposed to horizontal frontend/backend/repository splits."?
But I'm still not sure. Wide integration testing is necesity for reliable microservices. It's difficult to do that with dependencies being out of the scope of the change.
Re integration testing: That's why things like contract testing are becoming more popular. You make sure that you do as much testing as you can without relying on an entire ecosystem of depdendencies being available, so once you DO release something you can be fairly confident that it's already working as expected.
Classic example in favor of monorepos is shared internal library. You should be able to change its API in one step, because that's the way to not having to care about compatibility at all. Compatibility concerns only happen if there's a time difference and with atomic company-wide update, this just doesn't need to happen.
In my example, backend and frontend can be deployed together. And we can discuss whether they should -- I would say there is little benefit of maintaining such single-purpose, wide interface just for getting the ability deploy frontend changes only, but this would depend on care the team put into configuration of cluster orchestration.
Contract testing is good, but not as good as testing against the real thing. Very much depend on the circumstances though.
That makes sense to me, to have 3 repositories. Common code, perhaps even in your own dsl. And then implementations done in the various platforms.
So far the only good reason I've had for separating them has been related to deployment concerns (e.g. your API is deployed to AWS and your browser client is on Cloudflare).
If your API is a separate product with a separate release cycle, proper testing and backwards compatibility checks, then different repo.
It depends. Are you planning more projects which may share functionality? Are you going to release within a company with other teams that may want to integrate? Are some components managed by dedicated teams who may have their own cycles of work? Are you planning to split off a specific service soon?
Or to quote risky business: what is the problem we're trying to solve?
If something only may be the case, then I'd say its premature effort to plan for something that may or may not happen. Better to do the simplest thing now and then refactor when the need arises.
All those direct interdependencies, combined with the impossibility of just using package versioning to allow a library's consumers to upgrade at their own pace, meant that developers had a strong disincentive to do any cleanup that might result in a breaking change. They'd tend to prefer patches instead. So the long-term trend was for everything to eventually become a tangled mass of patchwork.
Google has entire teams devoted to combatting this problem. I'm not sure most companies can (or want to) afford that.
This is one of those spots where I think it's wise to be cautious about blog-driven development. By analogy, I've seen experienced woodworkers have fun with mocking the DIY furniture projects that come up in places like Pinterest. One common criticism was that, since wood expands and contracts with changing humidity, but only in certain directions, if you put it together in certain ways, you end up with a product that will literally tear itself apart over time. Of course, this isn't apparent from the Instructable article, because it's a slow process, and all those pictures were taken when the piece was still brand spanking new. It's also not something you're likely to see written in the comments section or anything like that - the people I've known who know this sort of thing aren't generally the type to write Instructables articles or maintain their own DIY blogs.
I suspect something like this also happens with any decent sized codebase: Code is a living thing that expands and contracts with changing business requirements. Over time, it will twist itself out of shape. Weak coupling and fragmented codebases are two different (though related) ways of trying to limit the damage from this effect. They both bring their own challenges, though, and aren't perfect. Monorepos don't have the same drawbacks, but that doesn't mean they don't have any. One, I think, is that you're left directly exposed to the full strength of the forces that cause bit rot.
If you're disciplined enough to recognize this and treat it as another incentive to attack bit rot head-on, then you'll probably end up in a better place for it. If you think monorepos are a panacea, though, you're setting yourself up for the kinds of failure that people generally don't like to blog about.
Genius phrase. Overall, that might be one of the best comments I've ever read on HN. The DIY furniture analogy is a good one. There's so much focus on what seems to work today, and so little awareness of what affects long-term maintainability. I see this at work too, everyone chasing "impact" for the half even when it screws us over any longer time scale. Understanding how code evolves over time is an essential part of being a competent developer. It's much more important than any in-the-moment coding test.
When I discovered this, I asked the module's maintainer about it. He explained that it used to be used, but requirements changed so that it wasn't needed anymore. He decided to leave it anyway, though, because the alternative would have been an immense effort to track down all the call sites, remove the argument, then trace all the resulting dead code, and remove that as well. It would have eaten up a week, I'm sure. Everyone had bigger fish to fry, and always would have bigger fish to fry.
As I understand it, Google gets around that problem by having a group of people where frying fish like that is their primary responsibility.
So the reason it happens in a monorepo more is that those connections are easier to make and therefore a lot more common, not because those connections are impossible in a multi-repo setup.
There's a number of solutions to prevent tight-coupling.
1. Language-level visibility modifiers. 2. Build system visibility. Bazel offers fine-grained visibility. I'm not sure about others. 3. Using RPCs as the primary interconnect between services.
My view is that disadvantages of tight-coupling in a mono-repo is a much better problem to have than the disadvantages of having logic spread across different repos.
There are legitimate arguments for both options. The thing I'm trying to say is, you should beware the idea that going monorepo means the ocean's going to just boil itself somehow.
I'd rephrase this: Google gets around this problem by having tooling (one such tool is the monorepo) that makes fixing issues like these fairly simple. A straightforward example like this one could be fixed with sed/clangtidy/refaster (depending on the language) + Rosie[1] in a few days, even if there were thousands (or 10s of thousands) of uses. But the work of upgrading is borne by the person doing the upgrade, not client teams who suddenly encounter compile errors when they try to version bump. This is much "fairer".
[1]: https://cacm.acm.org/magazines/2016/7/204032-why-google-stor...
For a widely used library. Leave the argument. This argument could be defaulted to NULL, the function can be marked @deprecated and it can log a warning message that the argument has no effect (depending on the language). That's how to handle updates with backward and forward compatibility. No need to break user code.
That's a strength of a mono repo. It only lets you do the right thing. You can't ignore other developers in the company and break all their software.
In a typical multi repo, there is the luxury to change anything anytime, as often as a developer feels like renaming a variable or a parameter. Creating a ton of unnecessary work for other developers who have the pain to keep up. That is, if they ever attempt to upgrade any library, there is no benefit to upgrading and it's impossible to keep up with all changes going on in all libraries thus it's rarely done in practice. The more it's behind, the more work there is to upgrade, the less likely it is to be done.
Not sure this is what people mean by a monorepo ...
[1]: backend → https://github.com/SHxKM/django-vue-ssr
[2]: frontend → https://github.com/SHxKM/django-nuxt-ssr-front
[3]: live-app → https://django-nuxt-ssr.herokuapp.com/ (on a free dyno, could crash on first load)
I've also forgone proper IDE "project" in favour of opening the directory in my editor on an ad-hoc basis from the command line (e.g. `cd frontend && subl .`). It's not perfect, but it avoids the issue of having too many windows, or having to fumble through IDE project switching dialogs.
Maybe I'm missing something, but couldn't you get pretty far with something like:
# This tells the tool where to look for changes
packages_folder: ./packages
# List dependencies. In this instance, if any files in `shared_api` change, frontend and backend need to build.
dependencies:
- frontend:
- shared_api
- backend:
- shared_api
# Definite build order
build_order:
- shared_api
- backend
- frontend
# Somehow find these scripts in each package subfolder and try to run them.
scripts:
- build
- test
- deploy
One disadvantage: a lot of IDEs don't cope well with monorepos as they try to index the entire thing.
There are several smaller/micro services that are still separate, however. They're generally less effected by the coordination of db/api/ui changes.
-depth=1 ameliorates this somewhat (though plays poorly with tag-heavy workflows).
They could have solved the pain by adding more repos as well. They never considered this as far as I can tell.
> We had three choices: ... Extract Core out of PSPDFKit-Android and again deal with fragmented pull requests.
Adding more repos would once again split up pull requests for features that touch multiple codebases, which is the reason they merged two codebases into one repo in the first place.
I am assuming they did copy/paste coding to get features into both. It is possible that they actually did write everything twice, but that is generally such a stupid idea I can't believe it (except for UI - it makes perfect sense to write a the UI for android and iOS completely separate - the logic behind the UI should still be common, but the UI elements/APIs are different enough that common probably isn't worth it)
Adding more repos should not split up pull requests. If it does that is a function of bad architecture.
It would be a simple one that could check the git log and see what files has changed since last time.
Or check the hash of each folder and compare it to previous commit.
If you use Docker, you can leverage that docker will take a hash of the context and if the images already exist on the machine use the cache if nothing has changed for those files. That is also quite effective +1
To me it feels like a misguided attempts to work around the need of having a package manager
Why not simply build your SRPM or DEB package repo for code? Worked like magic for me on four of my past workplaces.
This also works towards building your culture of stable releases instead of non-stop development and hotfixes.
The outcome of multiple repos should be modular libraries.
Yes; also they should all be treated as separate, independent products. Where multi-repos go wrong is when a whole product spans multiple repos, and a feature of the product depends on features being added or altered in multiple projects.
Ideally, the split should be so that nobody has to work across multiple repos to finish a feature.
The github / NPM ecosystem is IMO a good way to look at it; libraries that are all independent of one another who have a (usually pretty good) adherence to semver.
That does not sound realistic. Often I have to add a new function, and I think: where (in what library) does this function belong, and very often it's not the main library I'm working in at that moment.
Microservices without network headaches.
Modular code in subdirectories is what gave microservices an excuse against the "monolith".
Mono repo makes violation of modular rules easier, just link it in - in a complex project odds are nobody will notice. However this doesn't mean monorepo is bad, it just needs that you need to be careful to ensure the rules are written down.
Multi-repo makes violation of those same rules harder - you have to make a bunch of changes to ensure that the bad dependency is in the environment. This hints at the downside of multi-repo: you need to have some sort of dependency management system to ensure that the right version is there when you build.
Sometimes a repo contains code written in multiple languages.
IME it solves the same problem that monorepos are intended to solve, but without any of the downsides.