Why Google stores billions of lines of code in a single repository (2016)

Wouldn't the right tooling be able to show you changes to the slice of code you're interested in? I remember SVN would allow you to checkout just a single subdirectory, for example.

> all sorts of operations take longer (pulling, grepping source, etc.) to support code I couldn't care less about.

Makes sense about the pulling, but again, wouldn't grepping be configurable to only search where you need to?

> Frequently have to update the world at once. Unless the repo can store multiple versions of the same module, then all the consumers have to be updated at once, even if it's inconvenient. Sometimes migrations are better done gradually.

I'm not sure I understood you here, can you expand on that? Do you mean all the devs have to update their module? Why not use tagged/branched version of libs instead of working off trunk?

What kind of tooling do you use for the distributed approach?

To upgrade format G2 we must change both software A and B.

First, software B version 2 must accept both G1 and G2. To do this we may need to build software A version 2 and try them in a sandbox environment to gain confidence that ∀F1 we produce the correct G2. If F1 is complete, we may be able to do this exhaustively, but if F1 is sufficiently diverse, monte carlo simulation might be used.

Then, if there's a 1:1 relationship between A/B we can upgrade pairs.

If there's a N:M relationship, we need to upgrade all of the instances of software version B1 to B2 (at least within a shard). If you're running in a non-stop environment, this might have it's own challenges. Only then, can we begin the upgrade from A1 to A2.

Now:

Something, somewhere needs to record what and where we are in this journey. It is relatively straightforward how to do this with a monorepo, but it is very unclear how to do it with a distributed repository:

Almost everyone I know punts and uses some other golden record (like a continuous integration server, or a ticketing system, or an admin/staging system), and like it or not: that's your monorepo.

Note that even where Google are forced to use git (e.g. Android, Chrome) they use a many-repo approach.

Git only allows to check/commit/view to the entire repo at once. Then, some git operations are superlinear with the number of files or revisions, they are slow on large repo to the point of being unusable.

It's mandatory to have operation on a per file or subdirectory level in a mono repo approach. Companies that have mono repos all built tooling to support it. CVS/SVN used to do that out of the box but everyone hate them now.

I’ve never worked in a monorepo, so may be wrong, but this point presumes a dependency on “latest” at all times. I’d assume the components in the mono repo still release versions to the various package management systems (maven, pypi, npm, etc), allowing dependencies to be more stable.

Has that not been the common experience by those that have worked in them? I see a lot of merits of having to update everything at once (less code rot, hopefully) but it does seem to have drawbacks (many have commented on these as well).

I still do not see the appeal in mono repos since you're heavily dependent on discipline to not introduce spaghetti dependencies, where you fix one bug, but introduce 4 new in unrelated parts of the code. Now you solve that with an if statement, and thus we introduced a great deal of technical debt.

> 1) Difficult to track changes to the code I'm interested in.

What's wrong with 'git log $PATH'?

> 2) all sorts of operations take longer (pulling, grepping source, etc.) to support code I couldn't care less about.

A different format could help here, as can different tools (e.g. ripgrep or ag instead of grep). The time spent on those operations has to be balanced with the time spent updating your code to deal with someone else's incompatible library changes, again, when the other person is on vacation and you have no idea what the new philosophy of his library is. And you don't have any choice about updating, because another one of your dependencies that you really must update has already been updated to rely on his changes.

> 3) Frequently have to update the world at once.

IMHO that's a feature, not a bug. The person or team responsible for breaking the world is responsible for fixing it, rather than getting to break the world, then pop off down to Barton-on-Sea for an extended holiday while everyone else in the company gets to update his code to use an entirely different idiom.

> 4) Encourages sloppy dependency management.

My experience has been that multiple repos tend to encourage sloppy dependency management, while a monorepo tends to encourage deliberative, collaborative, professional dependency management. That's just my own experience, and of course different organisations will differ.

> I'm sure people will say "if you're having those problems, you're doing it wrong" but the same thing could be said to people who find the distributed model problematic.

My own experience has been that multirepos tend to be like dynamic typing and monorepos tend to be like static typing: multirepos can in theory be done right, but in practice they never are, while monorepos work, but at the cost of people having to colour within the lines. Which makes sense for any particular organisation may actually be a function of its maturity: if a place is trying to move fast and break things, maybe multiple repos make sense; if it's trying to deliver quality software, maybe a single repo makes sense.

3 and 4 are pretty fundamental though, especially 3 - if you don't want to force everybody to keep up with head, you probably don't want to use a monorepo.

I'm asking because i wouldn't know how to setup a mono repository at my 50 people Startup even if we deemed this to be necessary.

i.e. what stops your current tools from `for each repo, run...`, or how is monorepo fundamentally more capable than building automated library management / releases / etc with the same level of tooling?

Builds can (and usually do) depend on things that aren't part of your local checkout.

I'd say CitC is a much more accurate representation of the way Piper and blaze "expect" things to work.

For a huge, non-open codebase there are some pretty large downsides to a fully distributed VCS in exchange for relatively few benefits.

Until very recently there was a versioning system for core libraries so those wouldn't typically be at HEAD (minimizing global breakage). Even that has been eliminated now and it's truly just the presubmit checks and code review process that keeps things sane.

Another issue with git monorepos is access control, does anyone know of good solutions for this, does GVFS solve this also?

The same is true of svn, which many people like to bash nowadays, even in this discussion.

That way, you can still have a distributed repository (Git, Mercurial, etc.) if you want. Even if some code exists only in some developer's local repository, it's presumably not that big of a deal since that code can never have made it to production.

This doesn't require a single central repository, just that all repositories live in a common location.

> 2) Faster. To check out the source code for the project I now work on takes an hour in the distributed system, while it only took 5 minutes in the centralized system.

What distributed repository management system do you use, and what centralized system did you use?

> 3) Always backed up. All code that is checked into the central repository is backed up. It has happened twice that employees have left and code was lost because they only checked it in locally.

As with point 1, this doesn't require a single central repository, just that all repositories live in a common location.

This used to be true, but today these are all in fact the same hg repo (www as a possible exception, I'm unsure). The "sparse checkout" machinery disguises it, but for engineers working cross platform (e.g. React Native) it's routine to make commits that span platforms.

First, like in other areas, I see companies that want to "google scale" and blindly copy the idea of monorepos but without the requisite tooling teams or cloud computing background / infrastructure that makes this possible.

Second, I worry about the coupling between unrelated products. While I admit part of this probably comes from my more libertarian world view but I have seen something as basic as a server upgrade schedule that is tailored for one product severely hurt the development of another product, to the point of almost halting development for months. I can't imagine needing a new feature or a big fix from a dependency but to be stuck because the whole company isn't ready to upgrade.

I've read of at least one less serious case of this from google with JUnit

> In 2007, Google tried to upgrade their JUnit from 3.8.x to 4.x and struggled as there was a subtle backward incompatibility in a small percentage of their usages of it. The change-set became very large, and struggled to keep up with the rate developers were adding tests.

https://trunkbaseddevelopment.com/monorepos/#third-party-dep...

(All of it is kind of depressing when one's comparison is non-tech companies, though.)

This is indeed true, the various efforts to upscale git (msft gvfs, etc) run into this and try to upstream things, but it is slow going.

This was kind of cumbersome to maintain TBH, and the fact that changes to different repos can be dependent on one another seems to strongly suggest that the code should be together in the same repo. Personally, I opt for mono-repos until I'm forced to change for whatever reason.

For surface level changes this is often quite small. For changes to core libraries, well, you run a lot of tests.

$ bazel query 'rdeps(//foo/my:target, //...)'

Of course, this query in the monorepo will take a long time or not work, because the target universe of "//..." is far too large. This is where other systems come in.

[0] https://docs.bazel.build/versions/master/query.html#rdeps

To me, Piper is a monolithic version control system which is geared towards good engineering practices.

As far as I know there are only two such systems in use today and the other one is very dated and older than a lot of things out there.

When people say they have worked in a monolithic repo, then they typically mean a 1 repo under one of the open source version control systems, but none of these actually do or support what is needed when working with a monolithic repo AND modern/good engineering practices.

For that specialised VCS is required and there is very few examples of that, none of which is in open source software.

Git probably could be made to do this kind of stuff, but it would require some extensions to the DAG as well as extend on it's already verbose command line set. But I think it is doable.

The question is who can do it? Most are probably under some strict NDAs.

I think if you want to make sure that all your changes are in, you need to do what most programmers do and learn to finish a programming session with a commit and push, just like you finish a sentence with a dot. Once you are used to it, the chance of forgetting are really low.

I agree with regards to committing regularly, but sometimes life happens and one can forget.

I’ve been considering writing a python script that checks my local repos for uncommitted/unpushed changes and - now I think about it - perhaps also runs when I start a new terminal session just for good measure.

2. Having no friction to change anything makes you far more productive and ambitious.

3. Scripting at a org-level means you can automate things more easily and more in depth.

We run an entirely node stack so Lerna enables this in the first place. Given that, I'd never move to more than one repo if possible. It's almost all downside: more overhead/fragmentation, less control, more wasted time/mental overhead moving between things, API friction that reduces ambitious change.

Only downside of monorepo is Github not supporting them well. If you want to release some sub-packages as OSS, or want to use GH to track issues you're stuck using one big repo to handle everything. I'd bet Github fixes this within the next year or so though.

These sorts of small, general, large scale cleanup commits are quite common at Google, and they're encouraged. They help keep the codebase healthy. There are special groups that review them so that all of the individual teams affected don't have to bother, and there are tools to manage the additional testing and approval requirements for such a change.

At my previous company, making such a change would have been a major undertaking. I never would have considered a refactor of that scale without a critical need. They had thousands of packages, each of which had its own repository and an incredibly complex web of build and runtime dependencies. It was a nightmare, and fiddling to find a working sets of versions of internal dependencies took up way, way too much of my time each day.

With a fragmented large code base you're in a world of hurt because you're dealing with versioning. There is no guarantee of when every other dependency will migrate to the latest code path.

But again, if you're in a 1-10 person team working on some trivial codebase, a monorepo might not be helpful. If you have 500 engineers working on a single codebase, tradeoffs change.

[1] - https://buckbuild.com

Source: Googler.

This architecture assigns each line of code a nested history: the public commit log, and also the sub-history of each commit, which evolved during code review.

IMO it would be better if the code review changes were manifest in the public commit log (e.g. via feature branches), instead of being tracked separately. The code review layers add duplicative complexity.

when the CL is first uploaded or sent for review the OCL is assigned, the CL number is assigned on commit.

As for separation of concerns, there's actually a lot of scoping / visibility stuff precisely to avoid letting people depend on things they shouldn't. I think you now have to explicitly open up your visibility to let random other projects depend on you. A monorepo doesn't require that you share, it just permits it.

https://www.youtube.com/playlist?list=PLAD8A7B6D66DDD297

> Key insights

> Google has shown the monolithic model of source code management can scale to a repository of one billion files, 35 million commits, and tens of thousands of developers.

> Benefits include unified versioning, extensive code sharing, simplified dependency management, atomic changes, large-scale refactoring, collaboration across teams, flexible code ownership, and code visibility.

> Drawbacks include having to create and scale tools for development and execution and maintain code health, as well as potential for codebase complexity (such as unnecessary dependencies)

Dagger2 was invented by Google for good reason because it works purely as an annotation processor without runtime bytecode classloader magic. That means it is easier to debug, faster to startup on mobile, and can be crosscompiled with j2objc and GWT/j2cl.

Much of the time in-house stuff is created to deal with scalability or maintain ability problems.

I would love to have a monorepo, although I'd advocate for branch-based rather than trunk-based development. Yes, merges can be their own kind of hell, but they impose that cost on the one making breaking changes, rather than everyone else.

It fulfills a lot of the same goals of having a monorepo in terms of scaling a large organization, while having somewhat different pros and cons.