I tend to much prefer splitting out reusable packages into their own repos with their own packaging and unit tests and tagging to whatever version of that package. It makes it MUCH easier for someone to work on something with minimal overhead and be able to understand every line in the repo they are actually editing.
It also allows reusable components to have their own maintainers, and allows for better delegation of a large team of engineers.
I went from many years at FB, to a place like you describe - hundreds of small repos, all versioned. It’s a nightmare to change anything. Endless git cloning and pulling and rebase. Endless issues since every repo ends up being configured slightly differently, and very hard to keep the repo metadata (think stuff like commit rules, merge rules, etc) up to date. It’s seriously much harder to be productive than with a well-oiled monorepo.
With a monorepo, you wanna update some library code to slightly change its API? Great, put up a code change for it, and you’ll quickly see whether it’s compatible or not with the rest of the whole codebase, and you can then fix whatever build issues arise, and then be confident it works everywhere wherever it’s imported. It might sound fragile, but it really isn’t if the tooling is there.
Tooling isn't the problem though, the problems are:
- multiple monorepos copying code from each other, despite that code should be a library or installable python package or even deb package of its own
- you will never understand the entire monorepo, so you will never understand what things you might break. with polyrepos different parts can be locked down to different versions of other parts. imagine if every machine learning model had a copy of the pytorch source in it instead of just specifying torch==2.1.0 in requirements.txt?
- "dockerize the pile of mess and ship" which doesn't work well if your user wants to use it inside another container
- any time you want to commit code, 50000 people have committed code in-between and you're already behind on 10 refactors. by the time you refactor so that your change works, 4000 more commits have happened
- the monorepo takes 1 hour to compile, with nothing to compile and unit test only a part of it
- ownership of different parts of the codebase is difficult to track; code reviews are a mess
- dont have multiple monorepos
- use blaze or similar and run all downstream tests, the binary for your ml model includes pytorch whether you build from source or requirements.txt.
- other people committing doesn't matter if you are isolating presubmits and similar.
- using a blaze-like you never compile the whole monorepo
- code owners etc. makes this straightforward.
Like, as someone whose career has been mostly at Google, these are not problems I encounter at all, or only in cases where you'd have similar scope of problems no matter the repo structure.
Could you elaborate on this one?
It's only a problem if someone submits a file you're touching, in which case you just have standard merge conflict issues.
Google3 is a true monorepo. 99% of code in the company is one repo, with minor exceptions for certain open source projects and locked down code.
Edit: for example, you can change YouTube code, Google Assistant code, a Maps API, some config for how URLs are routed at top level load balancers, etc all in one CL if you really wanted to/needed to.
This works well for a couple dozen repos per team in my experience. It’s also my preferred way to work.
It doesn’t scale so well to hundreds of repos per team without significant tooling. At some point anything cross-cutting (build tool updates, library updates, etc) becomes hard to track. Repos are left behind as folks change teams and teams are reorg’d.
I’ve never worked in a monorepo, but I can see the appeal for large, atomic changes especially.