Case Study: Monolithic Repository at Google (2018) [pdf] | Hacker News Reader

Case Study: Monolithic Repository at Google (2018) [pdf]

people.engr.ncsu.edu

113 pointssvat32 comments

This paper from last year surveys Google “engineers who have experience with both monolithic repos and multiple, per-project repos” to see what exactly they like and dislike about monorepos. The results match my experience, and I find the paper well-written (especially like the “Threats to Validity” section that says what biases may have affected the results; more papers and arguments should have them IMO).

Older reading on monorepos that I'm aware of:

• Dan Luu, Advantages of monorepos (2015) http://danluu.com/monorepo/

• Why Google Stores Billions of Lines of Code in a Single Repository (2015): https://cacm.acm.org/magazines/2016/7/204032/fulltext (HTML) / https://cacm.acm.org/magazines/2016/7/204032-why-google-stor... (PDF) / https://www.youtube.com/watch?v=W71BTkUbdqE (video)

This paper is from the “Software Engineering in Practice” track of the ICSE 2018 conference:

• https://www.icse2018.org/track/icse-2018-Software-Engineerin...

What engineers like the most:

• “I can search across our company's source code”

Interesting point: “Reduction of cognitive load” and “available developer tools” are cited as advantages of both monorepos and multi-repos.

Yes. There are different kinds of programmers, different way of thinking. Just as there are personality tests, there could be tests to identify what type of developer you are to select a set of tools and configurations that are better matched.

Would it be naughty (legally) to discriminate based on the results of this test?

>In both cases, the related tooling is also a significant factor; engineers favor particular tools and are drawn to repo management systems that support their desired toolchain.

>Like virtually all choices in computing, the choice of whether to use a monolithic or multi-repo model is a comparison of tradeoffs.

https://github.com/korfuri/awesome-monorepo

This topic is going to keep coming up because there isn't a single correct answer for all situations. It's like arguing about which editor is better.

I have my opinions, but at the end of the day you need to choose what is right for your team and your project. You also should be open to new ideas and looking for ways to improve your current development processes and tooling. At the end of the day, version control is a means to an end and shouldn't be something that consumes a significant amount of your developers' time.

I wonder if there is some fertile ground for tooling that can bridge the gap so you don't have to make and actual binding choice when you setup your project. For example tooling that federates multiple repos in a way that preserves most or all the advantages of the monorepos.

For a company like google or facebook where everything is done "in house" this could be feasible. They had to create a lot of tooling to accommodate theirs engineers. But for a smaller company using tools like git, intellij, VScode, jenkins, etc and the tools for deploying your application doesn't the size and the complexity (number of files and their structure) eventually becomes a problem ? What's the size of their repo ? And at which point it becomes too much for most software to handle ?

So how does it work if a library owner changes something in his library—do they have to update all projects that depend on it and then push this as a single commit?

Yes. And that is easy because the tooling can find all callers and the test infrastructure can find and run all related tests (even, if need be, literally every test).

Generally it's on the library owner to update all projects that depend on it, i.e. not break their tests, but this need not be done as a single commit. More concretely:

If the change is internal to the library, i.e. does not require the projects that depend on it to change the code, then the library owner simply tests that none of the affected tests break (globally), and updates their library.

If the change requires dependent projects to change their code, then some way is first figured out of having code that would work with both versions of the library, then updating all the callers individually (e.g. in a separate mostly-automated change for each project to review), then we're in the first situation above.

“Old APIs can be removed with confidence, because it can be proven that all callers have been migrated to new APIs.” (from the older article)

I myself love monolithic. But the CI is a mess.

I'm using Jenkins. So right now I don't have a way to force build specific directory :(. It build entire thing :(. Anyone knows how to solve these problem? I have a Rails app, Go services, iOS, Android all in same repo..

It's interesting that we're moving into a microservice architecture but we try to consolidate the code base into one.

Wouldn't git modules solve your problem? You'd have to have multiple repos for each service though.

IMO this is the worst of both worlds.

Nice explanation

Gitmodule is really sucks when we have to bump a bunch of them.

We do this inside a Jenkinsfile, slightly annoying if your actual pipeline is broken and needs editing as you also need to do a fake edit to the application, but works 95% of the time for us.

def databaseChanges = sh returnStdout: true, script:"git diff --name-only HEAD^ HEAD database"

def etlChanges = sh returnStdout: true, script:"git diff --name-only HEAD^ HEAD etl"

def apiChanges = sh returnStdout: true, script:"git diff --name-only HEAD^ HEAD api"

Assuming each one of the rails app, go services, etc, are in their own subdirectory, hash the contents of the subdirectory and use that as one of the tags that is stuck on the artifact. Look up the tag in your artifact store. If it doesn't exist, run your build scripts.

wow, haven't though of it. Sound like very good strategy. Going to try it out.

That's interesting. In my experience, one of the major advantages of a monorepo is precisely CI. In fact, only after we tried a Monorepo we finally experienced true CI.

An example to illustrate this: You have a library X that depends on Library Y and application A that depends on X. Then when you make a change in Lib Y you want to make sure that the change doesn't break X or A. If A, X and Y live all in separate repositories with separate CI pipelines, then this is hard/cumbersome to solve. In a Mono-repo, if you use a build system like Bazel, it's very easy.

I believe when you say that your Jenkins setup is a mess, we experience a similar situation with GitLab CI. But I think it's important to realize that CI stands for "Continuous Integration". And running pipelines for every individual repository is not really testing integration, it's just testing an individual component. The solution is to use Bazel or similar. Then you will have proper CI.

We have multi repos, in fact dozens of them, that make up our public facing mobile apps. Our CI is a disaster as we have way too many simultaneous releases in flight at the same time (up to 7), not sure how this would even work in a monorepo.

I actually have easy deployment with monoliths with Jenkins in dot net. Deploying, creating nugets, ..

And conditional deploying ( eg. Update external libraries only.

I'm production, it's not continuous, but in beta and alpha it is.

You could have it check if those directories changed since the last build. Then it only builds what youve recently worked on

During my 18 months at Google circa 2015, I used the following repos: google3, Android, ChromeOS, and Chrome browser. 1 is 4 for large values of 1, after all.

One telling question is whether there have been any efforts to unify these repos. Monorepos can come about passively, through uncontrolled growth. But if you truly believe in the monorepo, surely you'll knock down the barriers and unify the distinct repos. Has Google done that?

If they've actively chosen not to, why not? Is Android separate from google3 for hysterical raisins, or on sound principles, or some mix? Is there a cross platform repo for shared code?

The last three are open-source, so doesn't it make sense to keep them out of the google3 monorepo?

Yeah no. It's because the development model is part of the product for things like chromium and Android.

You can add to your list a number of other codebases; kubernetes has it's source of truth at GitHub, for example.

These all add complexity, and not in a good way. With enough small repos you lose the ability to centrally monitor dependencies and make sweeping fixes. For example it becomes difficult to do things like safely patch your entire codebase all in one go to defend against the next embargoed vulnerability, where you have weeks or days to deploy a fix to all your products before it goes public.

But k8s does live inside third party. And then there's stuff like tensorflow, where the source of truth is in the monorepo, but PRs via github are accepted and merged in.

It's mostly historical reasons, and that they gain relatively little from merging them, but it costs a lot. No one in Google3 depends on Chrome or Android source, and nothing in the Chrome or AOSP projects depends on Google3.

Android and chrome are both on weird git systems, and that doesn't work well in Google3. There'd be a huge amount of work to merge them, and for what...exactly?

Is it possible to break this down into a set of generic pros-and-cons?

I have only good experiences with it but I think it could be unmanageable if there is no automated integration testing (with reverts.) That is, everyone's submits are subject to a build (if the build fails, your submit is automatically reverted) and a battery of tests (if a test fails, your submit is automatically reverted). So you have to be able to submit code changes and test changes as one whole change list.

I wanted to see an architecture diagram :’(

Reply on news.ycombinator.com