Google stores billions of lines of code in a single repository (2016) [pdf]
dl.acm.org
dl.acm.org
1. The single version dependencies are asinine. We are migrating to a monorepo at work, and someone bumped the version of an open source JS package that introduced a regression. The next deploy took our service down. Monorepos mean loss of isolation of dependencies between services, which is absolutely necessary for the stability of mission-critical business services.
2. It encourages poor API contracts because it lets anyone import any code in any service arbitrarily. Shared functionality should be exposed as a standalone library with a clear, well-defined interface boundary. There are entire packaging ecosystems like npmjs and pypi for exactly this purpose.
3. It encourages a ton of code churn with very low signal. I see at least one PR every week to code owned by my team that changes some trivial configuration, library call, or build directive, simply because some shared config or code changed in another part of the repo and now the entire repo needs to be migrated in lockstep for things to compile.
I've read this paper, as well as watched the talk on this topic, and am absolutely stunned that these problems are not magnified by 100x at Google scale. Perhaps it's simply organizational inertia that prevents them from trying a more reasonable solution.
Perhaps that might be the default case, but the build system has a visibility system[1] that means that you can carefully control who depends on what parts of your code.
Separately, while some might build against your code directly, a lot of code just gets built into services, and then folk write their code against your published API, i.e. your protobuf specification.
How would multi-repo change this? A dependency updated, and code broke, and the new version was broken—but you update dependencies in multi-repo anyway, and deployments can be broken anyway. I don’t see how multi-repo mitigates this.
> It encourages poor API contracts because it lets anyone import any code in any service arbitrarily.
This has nothing at all to do with monorepos. Google’s own software is built with a tool called Bazel, and Meta has something similar called Buck. These tools let you build the same kind of fine-grained boundaries that you would expect from packaged libraries. In fact, I’d say that the boundaries and API contracts are better when you use tools like Bazel or Buck—instead of just being stuck with something like a private/public distinction, you basically have the freedom to define ACLs on your packages. This is often way too much power for common use cases but it is nice to have it around when you need it, and it’s very easy to work with.
A common way to use this—suppose you have a service. The service code is private, you can’t depend on it. The client library is public, you can import it. The client library may have some internal code which has an ACL so it can only be imported from the client library front-end.
Here’s how we updated services—first add new functionality to the service. Then make the corresponding changes to the client. Finally, push any changes downstream. The service may have to work with multiple versions of the client library at any time, so you have to test with old client libraries. But we also have a “build horizon”—binaries older than some threshold, like 90 days or 180 days or something, are not permitted in production. Because of the build horizon, we know that we only have to support versions of the client library made within the last 90 or 180 days or whatever.
This is for services with “thick clients”—you could cut out the client library and just make RPCs directly, if that was appropriate for your service.
> It encourages a ton of code churn with very low signal.
The places I worked at that had monorepos, you might filter out the automated code changes there to do automated migrations to new APIs. One PR per week sounds pretty manageable, when spread across a team.
Then again, I’ve also worked at places where I had a high meeting load, and barely enough time to get my work done, so maybe one PR per week is burdensome if your are scheduled to death in meetings.
In a multi-repo world, I control the repo for my own service. For a business-critical service in maintenance mode (with no active feature development), there's no reason for me to upgrade the dependencies. Code changes are the #1 cause of incidents; why fix something that isn't broken?
We would have avoided this problem had we not migrated to the monorepo simply because, well, we would have never pulled in the dependency upgrade in the first place.
> In fact, I’d say that the boundaries and API contracts are better when you use tools like Bazel or Buck
I'm familiar with both of these tools, and I agree with this point. However, you are making an implicit assumption that 1. the monorepo in question is built with a tool like Bazel that can enforce code visibility, and 2. that there exists a team or group of volunteers to maintain such a build system across the entire repo. I suspect both of these are not true for the vast majority of codebases outside of FAANG.
> The places I worked at that had monorepos, you might filter out the automated code changes there to do automated migrations to new APIs
Sure, this solves a logistical problem, but not the underlying technical problem of low-signal PRs. I would argue that doing this is an antipattern because it desensitizes service owners from reviewing PRs.
https://www.usnews.com/news/health-news/articles/2022-10-10/...
https://www.bloomberg.com/news/features/2023-02-03/ireland-h...
https://floridapolitics.com/archives/585686-tallahassee-memo...
https://www.oceancitytoday.com/news/atlantic-general-hospita...
"critical industries like banking and healthcare".
Also what does "banking" mean anyway? That comment is so generic as to be meaningless. If you are talking about Internet-facing retail banks in 2023, most are very serious about security... because regulations, and giants fines when they get it wrong. And if the fines aren't large enough in your country, tell your democratically elected officials to 10x the fines. It will change industry behaviour instantly -- see US investment banks' risk taking after the Vocker Rule/Dodd-Frank regulations.
Firing people doesn’t get you un-hacked. When your risk model involves threats coming from the inside (and at sufficient scale and value it definitely should) then you want to harden things internally too.
This is now the “what is code rot?” discussion, which is an incredibly deep and nuanced discussion and I’m not going to do it justice here.
Just to pick an example—if you have an old enough version of your SSL library, it won’t be able to connect to modern web servers, depending on the configuration (no algorithms in common). If you have old database software, maybe it won’t work with the centralized backup service you have. If your software is stuck on 32-bit, maybe you run out of RAM, or maybe the vendors stop supporting the hardware you need. If you need old development tools to build your software, maybe the developers won’t be able to make changes in the future when they actually become necessary. What if your code only builds with Visual Studio 6.0, and you can’t find a copy, and you need to fix a defect now?
As much as I like the idea of building software once and then running the same version for an eternity, I prefer the idea of updating dependencies and spending some more time on maintenance. I advocate for a certain minimum amount of code churn. If the code churn falls too low, you end up with getting blind-sided by problems and don’t have any expertise to deal with it. My personal experience is that the amount of time you spend on maintenance with changing dependencies doesn’t have to be burdensome, but it’s project-dependent. Some libraries you depend on will cause headaches when you upgrade, some won’t. Good developers know how to vet dependencies.
If you really need an old version of a dependency, you can always vendor an old version of it.
If you can’t afford to put developers on maintenance work and make changes to old projects, then maybe those projects should move from maintenance to sunset.
> that there exists a team or group of volunteers to maintain such a build system across the entire repo
Bazel doesn’t require a whole team. My personal experience is that a lot of teams end up with one or two people who know the build system and know how to keep the CI running, and I don’t think this changes much with Bazel.
Bazel is actually very good at isolating build failures. You can do all sorts of things that break the build and it will only affect certain targets. It is better at this than a lot of other tools.
> underlying technical problem of low-signal PRs
I honestly don’t see a few low-signal PRs as a problem. PRs are not there to provide “signal”, they are just there to logically group changes.
The kind of PRs that I see go by in monorepos are things like “library X has deprecated interface Y, and the maintainers of X are migrating Y to Z so they can remove Y later”. Maybe your experience is different.
I do think that owners should not feel that they need to carefully review every PR that touches the code that they own. This is, IMO, its own anti-pattern—your developers should build processes that they trust, monitor the rate at which defects get deployed to production, and address systemic problems that lead to production outages. Carefully reviewing each PR only makes sense in certain contexts.
If you’re working in some environment where you do need that kind of scrutiny for every change you make, then you are probably also in an enviroment where you need to apply that scrutiny to dependencies. Maybe that means your code has fewer dependencies and relies more on the stdlib.
1) This is solved by 2 interlocking concepts: comprehensive tests & pre-submit checks of those tests. Upgrading a version shouldn’t break anything because any breaking changes should be dealt with in the same change as the version bump.
2) Google’s monorepo allows for visibility restrictions and publicly-visible build targets are not common & reserved for truly public interfaces & packages.
3) “Code churn” is a very uncharitable description of day-to-day maintenance of an active codebase.
Google has invested heavily in infrastructural systems to facilitate the maintenance and execution of tests & code at scale. Monorepos are an organizational design choice which may not work for other teams. It does work at Google.
Does this mean that some things will never get updated, as the effort required is impossibly high?
No to mention that if you're the first team to import a third_party library, you own it and other teams can add arbitrary cost to you updating it. You have to be very aggressive with visibility and SLAs to work around this.
If the third party dependency does not add enough value to justify the cost then don't add it.
This also means services can be left to rot for years because they don't need to be upgraded, while all the infrastructure changes around them, which is a giant pain when you do eventually need to change something.
If you have a multi repo architecture you absolutely need both clear ownership of everything and well planned maintenance.
Instead, some newer package completely replaces an old one, with no relation to the old dependency package, or with a dependency on some future one, and both can run at the same time while turning the old one off
If you're not responsible for fixing downstream dependencies then you don't need to spend any time figuring that out.
Deleting code that is not being used anymore happens way too rarely in my opinion.
Of course, the problem can be mitigated by a disciplined team that understands the importance of everybody being on the same page on which version of each library one should use. On the other hand, such a team will probably have little problem using monorepo in the first place.
Whether you have a monorepo or multiple repos, a good team will make it work, and a bad team will suck at it. But multiple repos do provide more ropes for inexperienced devs to tie themselves up, in my opinion.
If you have one key dependency update with a feature you need, but you need substantial code updates and 80 services depend on it, that may be impossible to pull off no matter what. Comparatively, upgrading one by one may not be easy, but at least its possible.
The importance of everyone being on the same page with dependencies might just be a limitation of monorepos rather than a generally good thing. Some services might just not need the upgrade right now. Others may be getting deprecated soon, etc.
Everyone claims different needs if they can. Nothing could be linked together anymore if you just let everyone use whatever they want.
Or maybe people start to try to workaround this by ... reinventing the wheels (and effectively forks and vendoring) to reduce their dependency graph.
There is a genuine need for single instance of every third party dependencies. It is not unique to monorepos. Monorepo (with corresponding batch change tooling) just make this feasible, so you don't hear about this concept for manyrepos, and mentally bind it to monorepo.
It certainly isn't for Java, hence why multiple classloaders exist.
For C and C++ it depends on the OS, on Windows (AIX, and similar OSes) this isn't an issue thanks to how symbol visibility works.
Two different libraries are free to have whatever versions they feel like.
That is how I managed back in the day to use JSF 2.0 on Websphere 6, which officially did not had support for it out of the box.
But I'm not talking about one binary here. I'm talking about multiple, separate services.
With Javascript it does not apply, alpha@1 can have its own gamma@1, beta@1 can have its own gamma@2. But the same does not hold for most languages.
left-pad is both amazing and sad. It's amazing because JS's "bundle entire dependency closure" approach, combined with npm infrastructure, successfully drove the usability of software reuse to the point that people even bother to reuse left-pad. This is beyond what a well-regulated corporate codebases can achieve (no matter strongly encouraged single instance or not, not matter manyrepo or monorepo), and it happens in open. It is sad because without being regulated people tends to do so too aggressively, causing, well, left-pad.
Got several thoughts on this one. First, lets look at how bad the issue really is:
To start using beta@1 you need to upgrade alpha@1 to alpha@2 that depends on gamma@2. What's the problem with that?
The same situation can arise with 3rd party dependencies, except there its much worse: you have zero control over those. Here you do have the control.
Now lets look at what this situation looks like in a monorepo: you can't even introduce gamma@2 and make beta@1 at all without
1. upgrading alpha@1 to alpha@2
2. upgrading all services that depend on alpha@2
3. upgrading all libraries that depend on gamma@2
4. upgrading all services that depend on gamma@2, if any
So you might even estimate that the cost of developing beta@2 is not worth it at all. Instead of quasi-dependency-hell ("quasi" because your company still controlls all those libraries and has power to fix the issue unlike real dependency hell) you have a real stagnation hell due to a thousand papercuts
My second comment is about building deep "layers" of internal dependencies - I would recommend avoiding it for as long as possible. Not just because of versioning, but because that itself causes stagnation. The more things depend on a piece of code, the harder it is to manage it effectively or to make any changes to it. The deeper the dependency tree is, the harder it is to reason about the effect of changes. So you better be very certain about their design / API surface and abstraction before building such dependencies yourself.
Major version bumps of foundational library dependencies is an indication that you originally had the wrong abstraction. No matter how you organize your code in your repos, its going to be a problem. (Incidentally, this is also why despite the flexibility of node_modules, we still have JS fatigue. At least with internal dependencies we can work to avoid such churn.) It should still be easier with separate services, however, as you can do it more gradually.
Last note on left-pad and similar libraries. They are a different beast. They have a clear scope, small size and most importantly, zero probability of needing any interface changes (very low probability of any code changes as well). That makes them a less risky proposition (assuming of course they cannot be deleted)
Those are my (hopefully nuanced) 2 cents.
The problem is the team maintaining alpha does not want to upgrade to gamma@2 because it's an extra burden for them, and they don't have an immediate need.
The debate is not about teams owning separate services, it's about teams owning libraries.
This is unavoidable if we are talking about FOSS, people should be able to do whatever they want, and they do. A company has an advantage here: you can install company-wide rules and culture to make sure people don't do this. Which, in this case, happens to be: let's keep a single version of everything unless you have really good reasons.
In this case, you still have the option of working with them to help them migrate to gamma@2, if the cost of maintaining gamma@1 is indeed too high and would negatively impact you in serving other customers. This was the original premise, wasn't it - upgrading all your dependants when you upgrade your library? That's still an option. The point is - you have more choices. And you can also help customers one by one - you don't have to do it all at once
I will agree though, restricting choices helps when the company is finding difficulty in aligning incentives through communication. But you do give up a lot for it - including ability to move fast and avoid stagnation.
From what I saw I'd say it's exactly opposite: allowing multiple versions actually means "make teams able to choose for stagnation". And because we are lazy, we certainly do! There is a non-trivial amount of people who believes "if it ain't broken don't fix it". I can work with them to migrate them over, but they might not want to do so! In this case, a hard "bump versions or die" rule is a must.
Maybe if you work in a small group of great engineers you don't need to set such rules and you can move even faster, but I unfortunately haven't found such a workplace :(
> you don't have to do it all at once
Yes. Nobody should do it all at once. Making "bump versions or die" compatible with incremental adoption is slightly harder (see sibling threads for how it's done). Still worth it I'd argue.
A big bang always sucks versus some migration over time
Upgrading the library broke many tests across the org, and no one wanted to own going in and getting each team to fix it. Eventually, the library had a v2 release, and people started to care about being able to use it.
Ultimately, they just forked the current release and appended a v2 to the package name.
Not the norm, but it happens. The monorepo works for Google, but I wouldn't recommend it for most organizations; we have a ton of custom tooling and headcount to keep things running smoothly.
From the mobile side, it makes it super easy for us to share code across the 50+ apps we have, manage vulnerabilities quicker, and collaborate easily across teams.
Does it? Or is it stopping Google from supporting products which only make millions in revenue because of the massive burden of continually updating?
Most products at Google are not dropped because the monorepo makes it difficult for them to support - and I'm not sure how it would or how you got to that association. Also, plenty of products that are killed are not in the monorepo.
They are usually dropped due to a mix of things, but a big part is just better product management.
The implemented product, as well as the vision for something like inbox or Google music is still way better than Gmail and YouTube music as the end user
>They are usually dropped due to a mix of things, but a big part is just worse product management.
Google is now at the point where their new projects fail (like Stadia) because they killed old products. Killing products has second-order effects.
Hmm, does that explain the golang module versioning requirement where v2 must have a different name?
> 3) “Code churn” is a very uncharitable description of day-to-day maintenance of an active codebase.
Also implicit in the discussion is the fact that Google and other big tech companies performance review based on "impact" rather than arbitrary metrics like "number of PRs/LOCs per month". This provides a check on spending too much engineer time on maintenance PRs, since they have no (or very little) impact on your performance rating.
Umm, from whatever I have seen in big tech "impact" is also fairly arbitrary. It all is based on how cozy one is with one's manager, skip manager, and so on. More accurate is "perception of impact".
Especially as it gets more and more nebulous at higher levels.
There are still a couple of things that develop on long lived dev branches instead of directly at head, but my personal opinion is the need for those things to do that is mostly overstated (and having sent them cls in the past, it's deeply annoying).
> The single version dependencies are asinine. We are migrating to
> a monorepo at work, and someone bumped the version of an open
> source JS package that introduced a regression.
> It encourages poor API contracts because it lets anyone import any
> code in any service arbitrarily.
> I see at least one PR every week [...] because some shared config
> or code changed in another part of the repo and now the entire repo
> needs to be migrated in lockstep for things to compile.
[0] Or allowed -- I last worked there in 2017.
Sure, but this is unsustainable. If service Foo depends on myjslib v3.0.0, but service Bar needs to pull in myjslib v3.1.0, in order to make sure Foo is entirely unchanged, you'd have to add a new dependency @myjslib_v3_1_0 used only by Bar. After two years you'd have 10 unique dependencies for 10 versions of myjslib in the monorepo.
At this point you've basically replicated the dependency semantics of a multi-repo world to a monorepo, with extra cruft. This problem is already implicitly solved in a multi-repo world because each service simply declares its own dependencies.
Not exactly unsustainable considering Google has been very successful with this approach!
Any smallorg probably wants a bare monorepo, git or what have you. If you grow to the point that becomes unwieldy, you can either invest in tooling the way Google has, or be prepared to split the repo into library and project repos in a way that makes sense for what your mediumorg has grown into.
A small organization of 1-20 people should not emulate the layers of tooling; just have a single git repo somewhere and call it a day.
This is absolutely no different from buildings. If you had to keep every building up to date with the latest building codes, you would tear them down way way way more often.
This is a really good point and I think accurate when it comes to smaller Google endeavors. I don't think this killed Stadia, for example, but maybe Google Trips (an amazing service that I don't think many folks used and likely had few development resources assigned, or none).
> Sure, but this is unsustainable. [...] After two years you'd have
> 10 unique dependencies for 10 versions of myjslib in the monorepo.
Note that this need for a policy exists for both mono-repo and multi-repo worlds. Handling of third-party dependencies ought to be independent of how the version control repository is structured.
> At this point you've basically replicated the dependency semantics of
> a multi-repo world to a monorepo, with extra cruft. This problem is
> already implicitly solved in a multi-repo world because each service
> simply declares its own dependencies.
That's why monorepos exist. They're not a way to force people to upgrade dependencies, and they aren't a get-out-of-jail-free card for thinking about inter-project dependencies. A monorepo lets you have a linear view of code history.
Phrased differently: many people approach monorepos as a way to force their view of dependency management on other people in their organization. The successful users of monorepos (including Google) take great efforts to let separate projects in the same repo operate independently.
you're imagining a situation and speculating up a problem that might occur in that imaginary situation. in reality, no one does the thing you said - you don't add random deps on random external javascript libraries that don't have sane versioning stories.
You can do this with a mono, as well. However, the conceit is that "in the same repo" means you can "change them together." It is very very tempting that "went out as a single commit" means that it went out fine. Which, just isn't something you see in a multi world.
In my experience with software projects, this is very much not the case. It's one of the main reasons I'm such a big fan of monorepos--I have been burned way too many times by the need to make atomic commits involving separate repositories.
Ideally, all tooling makes the separate nature of the projects transparent. They should test separately. They should deploy separately. If that is not the case, then yes, they should be in the same repo.
> However, the conceit is that "in the same repo" means you can
> "change them together."
If you're making a backwards-incompatible change to an API then you need to:
1. Make a commit to your library to add the new functionality,
2. Send separate commits for review by other teams to update their projects' code,
3. Wait for them to be approved and merged in, then merge a final cleanup commit.
If your repository is designed to enable a single commit to touch multiple independent projects then it's not a monorepo, it's just a single-project repo with unclear API and ownership boundaries.
I'd love for it to be a strawman. But I do keep finding them.
You do get me to question what a mono repo is. I've never seen one that wasn't essentially an attempt at treating a company as a large project. Akin to a modular codebase with a single build. Could be a complicated build, mind you. Still, the goal has always been a full repository build.
The third party documentation is public, one-version policies exist but they are exemptions.
I second your point about monorepo versus ball of mud. They are so different. And managing all of this is about social/culture, less science-y. If you don't have good culture around maintenance, well then, yeah, duh, it will fall apart pretty quickly. It sounds like Google spends crazy money to develop tools to enforce the culture. Hats off.
This prevents situations where "Gmail" ends up bundling 4 different, mildly incompatible versions of MySQL or whatever, and the aggravation that would cause. Or worse, in c++ you get ODR violations due to a function being used from two versions of the same library.
You can see this some with how obnoxious Guava was, back in the day. It seems a sane strategy where you can deprecate things quickly by getting all callers to migrate. This is fantastic for the cases where it works. But, it is mind numbingly frustrating in the cases where it doesn't. Worse, it is the kind of work that burns out employees and causes them to not care about the product you are trying to make. "What did you do last month?" "I managed to roll out an upgrade that had no bearing on what we do."
1, 2 and 3: Use separate dependencies for each package, so this doesn't happen. Use e.g. GitHub Actions or another CI/CD file filtering wisely: if a file is needed by two packages, tests for both packages needs to run whenever it's changed, before merging, in addition to usual end-to-end tests. Have vulnerable dependencies alerting and make sure to upgrade it everywhere it occurs.
2: Also have some guidelines on that and enforce it either automatically or manually in PRs.
If your organisation can’t work effectively within a monorepo then you should absolutely address the problem, either by fixing the problematic behaviour or by switching away from a monorepo. The problem isn’t monorepos, the problem is monorepos in your organisation.
IMO, it's more of a development paradigm rather than a mere technology. You cannot simply use monorepo in isolation since its trade-off is strongly coupled with many other tooling and workflow. Because of this reason, I usually don't recommend migration toward monorepo unless there's strong organizational level support.
When you're pinning on old versions of software it quickly turns into a depsolving mess.
Software developers have difficulty figuring out which version of code is actually being deployed and used.
When dealing with major version bumps and semver pins around different repositories that creates a massive amount of make-work and configuration churn, and creates entire FTE roles practically dedicated to that job (or else grinds away at the time available for devs to do actual work and not just bump pins and deal with depsolving).
In any successful team which is using many dozens of repos, there's probably one dev running around like fucking nuts making sure everyhing is up to date and in synch who is keeping the whole thing going. If they leave because they're not getting career advancement then the pain is going to get surfaced.
The ability to pin also creates and encourages tech debt and encourages stale library code with security vulnerabilities. All that pinning flexibility is engineering to make tech debt really easy to start generating and to push all that maintenance into the future.
My point is that in reality, we use what best matches our knowledge, experience and perception and prioritisation of the problems. I, for one, believe that a monorepo is dangerous for small teams because it encourages coupling - not only do I believe it, but I saw it with my own eyes. It also creates unnecessary dependency chains. Monorepos contribute to a fallacy that every dependent on an object must be immediately updated or tech debt happens. But that's not even remotely given.
In any case, companies like Google and Amazon have more than enough resources to deal systematically with the problems of a monorepo. I'm sure they have entire teams whose job it is to fix problems in the VCS. But for small teams I remain unconvinced that it is a good idea. We shouldn't even be trying to do the things the big guys do, unless we want to spend all our time working on the tools instead of our businesses.
Is this convention for monorepos to all share the same dependencies? Does monorepo imply monolith? Surely one could have dependencies per "service" for example a python app with its own pipfile per directory.
2) Private/public/internal modifiers
3) Independent builds/project in a monorepo
You're doing it wrong.
The point of monorepo is that if someone breaks something, it breaks right away, at build time, not at deployment time.
You're not really using a monorepo.
One problem is that a lot of developers at big companies code business logic into their modules/dependencies... So whenever the business domain requirements change, they need to update many dependencies... Sometimes they depend on each other and so it's like a tangled web of dependencies which need to be constantly updated whenever requirements change.
Instead of trying to design modules properly to avoid everything becoming a giant tangled web, they prefer to just facilitate it with a monorepo which makes it easier to create and work with the mess (until the point when nobody can make sense of it anymore)... But for sure, this approach introduces vulnerabilities into the system. I don't know how most of the internet still functions.
For instance in multi-repo environments I've often seen this pattern: own some code, bump an internal dependency to a new version, see it break, ask the person maintaining it what's us, realize this case wasn't taken into account, few back and forth before finding an agreement.
On the other hand in mono-repo environments, it's usually more difficult to introduce a wide changes as you face all consequences immediately, but difficulty is mainly a technical/engineering difficulty rather than a social one, and the outcome is better than the series of compromises made left and right after a big multi-repo change.
Compare that with hundreds of tiny repos, each with their own little dependency system. Testing a version bump across the board before mainlining it is much more involved and you are more likely to hit stuff in production which should have been caught in test.
The other two points sounds more like cultural issues which may touch on branch strategies, code review, and what's expected of a developer. Those mostly cultural issues that overlaps with technical are hard in a way that repository strategy isn't.
I don't believe this is true, except in the short term. Unless the writing party is guaranteeing you forward compatibility, your consuming code will break when you update.
This is (almost) the only reason API contracts are worth having; the reason doesn't go away just because you can technically see all the code.
1. Have some concept of visibility restriction e.g. Go language has internal package.
2. Ensure that every single package has a command to build the code.
3. Ensure that CI builds all the packages that changed our impacted by the change in a given pull request.
These three steps are mostly sufficient in having a monorepo. What you get in return is high code consistency and code visibility for the whole team.
Honestly their systems are almost identical. Amazon just creates a monotonically increasing watermark outside the “repo”. Google uses “the repo” to create the monotonically increasing watermark.
Otherwise, Google calls it “merge into g3” Amazon calls it “merge into live”.
Amazon has the extra vocabulary of VersionSets/Packages/Build files. Google has all the same concepts, but just calls them Dependencies/Folders/Build files.
Amazon’s workflows are “git-like”, Google is migrating to “git-like” workflows (but has a lot of unnecessary vocabulary around getting there - Piper/Fig/Workspace/etc).
I really can’t tell if the specific difference between “mono-repo” or “multi-repo” makes much practical difference to the devs working on either system.
Not sure how deployments and CD work at google but I think the picture is different at google for unit tests, integ tests etc. Amazon teams have more control over their own codebase and development practices whereas, based on what I know, google has standardized many parts of their development process.
It’s then “merged into g3” from that workspace.
“Merging to live” builds and tests all packages that depend on the update.
So for example, building the new JDK to live will build and test all Java packages in previous live, all of them need to pass their package’s tests, only then will the JDK update be “committed into live”.
The only difference is that Google runs all the presubmits / “dry run to live checks” in the CL workflow. Amazon runs them post CL in the “merge VersionSet” workflow.
If it is a new commit of an existing major version, then yes, existing code in live will be rebuilt before the packaged is committed to live.
Every week our pipeline would get stuck and some poor college grad would spend a few days poking around at Brazil trying to get it to build. Usually took 3 commits to find a working pattern. The easy path was always to pins all indirect dependencies you relied on- but that was brittle and it’d inevitably break until another engineer wiped the whole list of pins out and discovered it built. Then the cycle repeats. I worked on very old services that had years of history. I’ve often discovered that packages had listed dependencies that went unused, but no one spent time pruning them, even when they were the broken dependency.
At Google, I have no memory of ever tinkering with dependency issues outside of library visibility changes.
Amazon pipelines and versionsets and all that are impressive engineering feats, but I think a version-set was a solution to a problem of their own creation.
A good article to reference when this topic gets raised: http://yosefk.com/blog/dont-ask-if-a-monorepo-is-good-for-yo...
In the limit, there are only two options:
1. All code lives one repo
2. Every function/class/entity lives in its own repo
3. You accept code duplication
The problem though is that (3) is not a stable equilibrium. Most engineers have such a kneejerk reaction against code duplication that (3) is practically untenable. Even if your engineers are more reasonable, (3) style compromise means they constantly have to decide "should this code from package A be duplicated in package B, or split off into a new smaller package C, which A and B depend on". People will never agree on the right answer, which generates discussion and wastes engineering time. In my experience, the trend is almost never to combine repos, but always to generate more and more repos.
The limiting case of a mono repo (which is basically it's natural state) is far more palatable than the limiting case of poly-repo.
I really don't see how that would work for most companies in practice. Most of the off the shelf tooling used by companies with hundreds or thousands of developers assumes working with polyrepos. It's good we're seeing simpler alternative to Bazel but that's just one piece of the puzzle.
all of the stuff that you can’t do easily yet (vfs for repo, remote builds) just isn’t relevant enough at this scale.
I guess with a 1K engineering company you can afford a substantial build team.
Why Google Stores Billions of Lines of Code in a Single Repository (2016) - https://news.ycombinator.com/item?id=22019827 - Jan 2020 (121 comments)
Why Google Stores Billions of Lines of Code in a Single Repository (2016) - https://news.ycombinator.com/item?id=17605371 - July 2018 (281 comments)
Why Google stores billions of lines of code in a single repository (2016) - https://news.ycombinator.com/item?id=15889148 - Dec 2017 (298 comments)
Why Google Stores Billions of Lines of Code in a Single Repository - https://news.ycombinator.com/item?id=11991479 - June 2016 (218 comments)
Wait, that's an average of nearly 30 new files per commit. Not 30 files changed per commit, but whatever changes are happening to existing files, plus 30 brand new files. For every single commit.
Although...
> The total number of files also includes source files copied into release branches, files that are deleted at the latest revision, [...]
I'm not quite sure what this is saying.
Is it saying that if `main` contains 1,000 files, and then someone creates a branch called `release`, then the repo now contains 2,000 files? And if someone then deletes 500 files from `main` in the next commit, the repo still contains 2,000 files, not 1,500?
If that's the case, why not just call every different version of every file in the repo a different file? If I have a new repo and in the first commit I create a single 100-line file called `foo.c`, and then I change one line of `foo.c` for the second commit, do I now have a repo with two files?
I mean, if you look at the plumbing for e.g. `git`, yes, the repo is storing two file objects for the repo history. But I don't think I've ever seen someone discuss the Linux git repo and talk about the total number of file objects in the repo object store. And when the linked paper itself mentions Linux, it says "The Linux kernel is a prominent example of a large open source software repository containing approximately 15 million lines of code in 40,000 files" - and in that case it's definitely not talking about the total number of file objects in the store.
I don't think it's entirely clear what the paper even means when it talk about "a file" in a source code repository, or if it even means the same thing consistently. I'm not sure it's using the most obvious interpretation, but I can't understand why it would pick a non-obvious interpretation. Especially if it's not going to explain what it means, let alone explain why it chose one meaning over another.
> The total number of files also includes source files copied into release branches
I guess you haven't used Perforce or similar. a branch is a sparse copy of just the changed files/directories. they are not used very much.
> files that are deleted at the latest revision
so it means "one billion files have existed in the history repo, some are currently deleted".
> I don't think it's entirely clear what the paper even means when it talk about "a file" in a source code repository,
seems pretty clear - a source code repo has lots of files. at the most recent revision, some exist, some were deleted in some past revision. more will be added (and deleted) in later revisions.
it's very much not the same model as git.
hope that clears things up.
It certainly feels that way :-)
> > The total number of files also includes source files copied into release branches
> I guess you haven't used Perforce or similar. a branch is a sparse copy of just the changed files/directories.
Still not sure I see the distinction. Surely "sparse" or "not sparse" is an implementation detail. If I create a new branch in git, the files that are unchanged from its parent branch share the same storage, but the files that have changed use their own storage.
> so it means "one billion files have existed in the history repo, some are currently deleted".
I guess I'm struggling to understand what the point of this metric is? I get why "Total number of commits", "Total storage size of repo in GB/TB/PB", "Number of files in current head/main/trunk", or even "total number of distinct file revisions in repo history", could be useful metrics.
But why "number of files (including ones that have been deleted)"? What can we do with this number?
> hope that clears things up.
It's helping. Thanks.
Of particular note is that they published this many years after it had been shipped to their internal customers. This was not some position paper about "why we focus on ai" after not shipping any of their "breakthroughs".
I think building something that scales for one big repo is just a completely different problem than making it scale for a lot of small repos.
And after the layoffs, it's pretty clear that no matter how hard you work, you can get fired so what's the point in dedicating your career to something like this?
>How does it work?
Different projects are in different folders instead of different repos.
>I assume they're built separately and pushing code to one doesn't affect the other.
Yes, building or testing something only builds its dependencies.
However products are in the same repo such a gmail, youtube, search (frontend, mobile, server, infra, etc), photos, maps, play, translate and literally thousands of other internal and external products and projects.
How would you handle this situation as an IC? As a manager of one of the teams? As a skip-level manager of both teams?
As a budding IC on the team that wants the upgrade, you may want to go fix up the other team’s code for them so you can bring them along with the upgrade. Realistically, the further you get from Google’s level of engineering discipline and skill the more likely you are to encounter the following in the needs-1.22 codebase:
- horrible code that is hard to understand and therefore hard to refactor
- code with no tests, making it risky to refactor
- the team that wrote it have all left or been fired and no one is available to help understand it
- they are a remote team with no social relationship to you who interact entirely online, in writing, in the style of an aggressive subreddit mod
- deeply entrenched factions mean that even if you offer them a patch they will default refuse it because who are you to work on their codebase and they don’t need the upgraded numpy so why should they waste resources on reviewing something they don’t want
- misguided adherence to status enhancing terms like “audit” and “compliance” mean jobsworth ICs refuse to even look at your patch because someone somewhere once heard a friend of a friend whose company failed SOC2 because engineer from floor X made a change to code owned by floor Y and it went against policy
All of these social problems are real ones I have encountered and if you have solved these then you’re probably already happily in a monorepo already. If instead you work in an org full of teams pointing guns at each other in a fight to the death to stop any kind of cross org collaboration from sullying the purity of the tribal system then know this: it gets better, and if you build the right social connections then the technical efficiency of having your monobusiness executing its monomission inside a monorepo is within reach!
*bug
Still, I have recently hit a major issue with the fact that GIT (and other common version control sw) don't have per-directory ACL.
Has anyone dealt with this issue? Which VCS / configuration have you adopted?
The software is built daily, and everyone must be on the same version of every library.
Under the hood there are a bunch of repos, and there are exceptions, but largely operates as a monorepo.
> Access to the whole codebase encourages extensive code sharing and reuse [...]
Doesn't this strategy result in a great risk of massive code leaks from rogue employees? Even if read access are logged and the culprit found, it's too late once it's been published.
But I found this discussion on HN.
Security-by-obscurity isn't something to rely on (again, except in the case of things like abuse detection where there's no alternative).
If they chose to make their code invisible to me then I can’t fix it.
There you go, PDF free version.
This is sometimes a problem for open source dependencies, though, as there isn't always anyone whose job it is to keep them up to date. Some amount of NIH syndrome is because reinventing the wheel can be less work than integrating an existing wheel that was designed for a different vehicle with different specs.