Why Google Stores Billions of Lines of Code in a Single Repository (2016) | Hacker News Reader

Why Google Stores Billions of Lines of Code in a Single Repository (2016)

cacm.acm.org

112 pointstosh121 comments

Seems like this gets re-posted every few months.

We don't actually have a single repository, although Google3/Piper really is huge and dominant. Android, Chrome, Fuchsia, and myriads of other projects don't exist in the monorepo (well, they are mirrored there) because the tooling and libraries there are really geared towards server deployment, not devices.

tosh and others do a lot of reposts, Wikipedia links, etc

karma farming?

stumbled upon it (again(?)) in a recent comment on "Is Git irreplaceable": https://news.ycombinator.com/item?id=21980556

and thought this might yield an interesting discussion

Why does it matter to you? Does this submission hurt you?

I’m simply pointing out why we get lots of repeat submissions.

Some stories are great and worth seeing every few years.

Your anger gave away the fact that you do the same thing:

https://news.ycombinator.com/submitted?id=Tomte

The one downside is that it creates a lot of noise on HN. It forces new stories off the first page quicker.

Also, stories about Jerry Lee Lewis’ marriage to a 13 year old isn’t HN, for example.

I'm not angry, actually. I just wonder why you're so invested in what I'm doing. I'm actually proud of it. More people should un-earth cool stories and submit them.

About Jerry Lee Lewis: I disagree. The sheer fact that moral compasses were so wildly divergent between America and England, of all pairs of countries, so recently, is quite on-topic, in my opinion.

Also, I think it's great that you found only one story you disagree with in the thirty stories my submission page shows. Clearly you approve of my selection.

> I’m simply pointing out why we get lots of repeat submissions.

That's explicitly okay, as the moderators point out all the time.

You joined the conversation. I mentioned another user. I quickly looked at your account once you became indignant. I scanned your last submissions page and saw you’re karma farming too and picked out the most egregious submission.

I didn’t say it wasn’t ok. I’m sure I do it occasionally.

I’m waiting for the first person to write a karma farming AI. Take great stories, determine a decay time, then repost.

“HN Karma Farming with Deep Learning (in Rust)”

> I’m waiting for the first person to write a karma farming AI. Take great stories, determine a decay time, then repost. > > “HN Karma Farming with Deep Learning (in Rust)”

This probably doesn't even need any AI, just some scrapping and scripting to determine when it's worth resubmitting.

Somebody should implement this, seems like fun.

Of course, if it's done with deep learning... then it would be something else. Maybe even determine correlation with other themes in the other news of the day/front page, or month, etc.

Machine learning is just a currently popular approach that is called “AI”. Expert Systems are also AI, and a title that called a karma whore bot AI wouldn’t be technically misleading...?

You’re forgetting that people actually have to upvote submissions. Presumably not everyone sees every submission on HN every time thus are candidates to upvote it.

Sure, write that karma farming bot. Seems like a good service to those of us who want to see good content we may have missed, generate new discussions on classic content.

Complaining about “karma farming” has always seemed like a weird jealousy thing to me. Where you presumably care so much about karma that you’re annoyed over your arbitrary concept of karma-fairness. Who cares? If someone can get karma by reposting classic content for HNers who missed it the other times, more power to them. They reposted the content to an audience that wanted to see it. You can’t always take it personally or lash out at others just because you’re not part of that audience.

I’m not complaining. I’m providing deeper insight into why people are seeing repeated posts.

I did point out a downside, but have at it.

Why are the motives to Karma farming-- is it the fuzzy feeling of internet points? Or is it people wishing to gain a reputation with a community, and then leverage the account to share their world view / interests / etc. I suppose it could be a matter of both, but since points take a more minimal role on HN, it's tough to say why Karma farming would make more sense here versus other platforms.

I assume you're comparing to Reddit. I wouldn't say internet virtue points play any less of a role here.

Your total "internet virtue points" here are displayed in the top right corner (whereas on Reddit you have to go looking). The virtue points any given comments gains are not visible to other people (unlike on reddit) but losses are. That doesn't seem like a big net difference between the two platforms.

HN's karma system is just as good as Reddit's at faking a consensus and marginalizing whoever has even a slightly less popular viewpoint on a subject.

In a well-designed karma system, karma farming is something to encourage because it means users are posting content that helps the website.

(I don't know many well-designed karma systems, but, that's at least the goal. The feedback cycle exists for a reason.)

I have five figures worth of karma. I don’t see what it gets me exactly or why anyone would want to farm it, but I guess there are all kinds of different hobbies.

On HN, a small number of reposts is ok if a post hasn't had significant attention yet in the last year or so. This is in the FAQ: https://news.ycombinator.com/newsfaq.html.

It's true that repostiness per se is annoying, but it's more annoying for good submissions to miss out on attention because /newest is something of a lottery. Allowing a small number of reposts is a way to mitigate that randomness, until we can maybe work out a better solution.

Could we have this pinned because the whole "Google uses a Monorepo and so should you" has become a huge meme and grown legs.

And you're right this topic is on a circular timer.

imho the problem space of evolution + stability of systems is becoming more zeitgeisty

(more people are aware of the challenges and stumble upon monorepo as a tool to deal with this among other things)

  * unison
  * clojure (immutability, spec, …)
  * blockchain
  * microservice architecture

I mean honestly, having worked with a monorepo here, I'd definitely consider it for other places. It has advantages.

The only arguments I have found against monorepo that don’t seem wrong are “the developers aren’t that good at git and can’t handle it.” Other wise it’s just making it harder to find out what code is where.

Well Google's monorepo is not git, and not even a "standard" one because it's not just a VCS: it's also a storage and a build infra, all in one. So while it's true that many developers wouldn't be comfortable using monorepos in git, it's not the right criticism. The real one is "there is no unified tool to do it correctly". Or at least I don't know of any.

Another thing your examples of projects developed outside of the main repo have in common is that they're open source.

When I was working on an open source project at Google (mod_pagespeed) we initially developed it in the main repo with exports to GitHub. Then we moved our source of truth to GitHub to make it easier to work with external contributors.

(Speaking only for myself, not for Google)

Not all are open source, I just refrained from referring to internal projects.

My understanding is that >99% of internal-only projects are in the main repo. If you think I'm missing something, perhaps write me internally (same username) and if I'm wrong I'll post here saying so?

Almost everything I work on hasn't been in the main repo for the last 5 years. I work in the hardware PA, and before that Fiber. I mean, it's all mirrored _into_ there, but work happens in separate git repos, using gerrit.

Prior to Fiber, yes, I worked in Google3. I've been @ Google for 8 years.

That still sounds like a monorepo. Google is using external projects and has public facing projects so isn't mirroring the way you should handle it? In what way could it be more monorepo like?

These projects are mirrored _into_ Google3, not the other way. And not all projects that aren't in Piper/Google3 are external facing. In fact only a minority of them are. I'm just not going to name them, for obvious reasons.

Mirroring only allows dependencies in one direction. Let's say package A is only in the monorepo and package B is mirrored into the monorepo. A can depend on some snapshot of B, (of, say, a client library) but B can't depend on packages from A, because outside of the monorepo B has no knowledge of A.

Also, one main advantage of a monorepo is you can make changes across many packages at once. The canonical textbook example is changing the API of my service: I can find all the instances in the monorepo of where that API gets used and change both the API and update all the calling instances in an atomic operation. You can't (or at least, shouldn't) do that to mirrored projects because the mirror in the monorepo is downstream from the source of truth for the project.

If you make backwards-incompatibilities API changes like this, you also need simultaneous/atomic deploys across all services.

On the other hand, if you aren’t doing simultaneous/atomic deploys, then having a monorepo may actually be an obstruction.

Yeah, that's the part I left out because it does get complicated when it comes to releases. For simplicity's sake, let's say everything actually gets bundled into one binary and released together.

> bundled into one binary and released together.

In fairness, that's the canonical test for "should this be 1 repo"

Not always. Renaming a proto field is one common example.

Is it similar to perforce? If so then github is also a monorepo, but much much more plesant.

Cisco uses a monorepo approach for IOS-XR. All platforms exist within the same repo, but the build system picks up only what’s needed to build the required platform’s image.

Of course, the scale is much lower, since it’s a really a single OS.

"We don't actually have a single repository”

This article doesn’t claim that. Its title doesn’t imply it, and its second sentence denies it:

”today the vast majority of Google's software assets continues to be stored in a single, shared repository”

It seems to imply it. Maybe not deliberately, but it confused me at first.

You are being pointlessly pedantic. A lot of people still think Google has only one repo because I articles like this.

In the 90s, a startup called WebTV managed to negotiate and buy a perpetual SOURCE CODE license for Perforce (P4). WebTV built their own good version control system called "SourceDepot" on top of P4.

When Microsoft bought WebTV they got this license, which enabled them to cheaply roll out P4/SourceDepot to massive teams for whom it was a big improvement over what they had been using. With access to the source, they could modify it to their needs and scale away.

Google tried to buy a source license early on, but Perforce had learned their lesson from what happened with WebTV and wouldn't sell them one. That's how Google ended up doing elaborate projects for years to prop up and eventually replace P4.

Source : I worked at WebTV, Microsoft and Google.

> Most notably, the model allows Google to avoid the "diamond dependency" problem (see Figure 8) that occurs when A depends on B and C, both B and C depend on D, but B requires version D.1 and C requires version D.2. In most cases it is now impossible to build A. For the base library D, it can become very difficult to release a new version without causing breakage, since all its callers must be updated at the same time. Updating is difficult when the library callers are hosted in different repositories.

Except they don't entirely. As soon as the monorepo interfaces with the rest of the world (like with npm packages, let's say), the problem just gets shifted onto the person trying to import/update a package. Which is further hindered by policy that discourages more than N versions of any given package.

Not that I'm against the policy or how it works. It just isn't "avoided" precisely.

I wonder how Googlers decide what ends up in the monorepo and what is outside of it. Collaborative open source outside (e.g. Android, Fuchsia, Chrome, …)?

It's often the product team's decision. Anybody can spin up a Git repo when they need one. There's freedom there, but you miss out on a lot of tooling and healthy discipline by doing so.

Once you're in the mono repo, you have to play by its rules (well...should anyway). Like in the case of npm packages I mentioned above.

Thanks for the context!

Presumably google doesn’t use npm or other external package managers that download random things from the internet. Why would they?

It's difficult in one repo too, although for different reasons. To update a dependency you may have to fix every caller in one very large change.

Basically the more users a single dependency has, the harder it gets to update.

Definitely not easy (& might yield to friction / demotivation).

The flip side is that the monorepo (more easily than otherwise) allows the person doing the change to bear the burden of updating the call-sites (& users).

Thanks for pointing out the difficulty this introduces.

paraphrasing Perlis:

Programmers know the value of everything and the cost of nothing.

This seems to me, to be a package manner implementation detail. The packager manager I am most familiar with, npm, won't have an issue with the diamond problem, since node modules can install nested modules, allowing B and C to both depend on unique versions of D.

Effectively rendering B and C incompatible in sharing D objects and structures. Just because you technically can doesn't make it a good thing in software architecture.

There is a reason why most globally linked runtimes (e.g. C++ or Java) do not encourage such mechanisms.

In times of security fixes sometimes there is only one secure version but the whole ecosystem is built around the fact that older, flawed version can still be included. Another drawback is performance, npm install is amazingly slow. In fact package resolution is NP-complete: https://research.swtch.com/version-sat

I generally agree but the problem is even worse :)

Yarn is a bit more sophisticated and has a concept of "resolutions" which would be great for the security use case of forcing a particular version. Even if B and C need different major versions, say D@v1-with-security-fix and D@v2-with-security fix, yarn supports that too, since resolutions can be nested.

https://yarnpkg.com/lang/en/docs/selective-version-resolutio...

What happens when B and C need to pass each other an object from D that's been changed?

Don't try a monorepo unless you have a team of engineers who can work full time on maintaining the build tooling, enforcing good practices, and training the rest of the engineers on how to behave in a monorepo.

If your organization lacks the ability to enforce good practices, train engineers how to behave, and otherwise acculturate people to your ways then there is no particular style of repo that can save you from failure.

I work in a tiny team (basically 2 people), but we have to work on multiple small projects (around 200k LoC total) including few shared libraries on Java. We used different repositories before with all releasing rituals, etc. But I tried monorepo and "all use latest" and it worked surprisingly awesome for us. Changes across repositories are much easier to do than before.

A team of two people can work with pretty much anything. The hard part is figuring out how to work with 100 or 1000 software engineers who all commit to the same repo.

Indeed, maintaining the greenness of the build in a giant monorepo is extremely expensive.

It isn't clear to me that it is worth the investment, even for Google -- though it clearly world's "well enough"

It's only expensive if you don't have tests and CI. If you do have tests and CI, it's much cheaper to maintain one repo than three dozen smaller ones: garbage simply can't get into master. Note also, that if you want to ship high quality software, having a fragmented repo doesn't free you from the need to have tests and CI, if anything it leads to duplication of effort.

I currently live in a monorepo that has pretty good CI and test coverage. That means that I know every time I have work to do because something changed (because the tests go red). That's great. But I still have to do the work every time someone changes anything that affects me

You're doing it wrong: if _they_ want to change something that affects you, _they_ should be the ones doing the work to not break you. The situations where _you_ have to do anything more than a code review of such changes should be very rare. That's how it works at Google.

And besides, it's not like this gets any easier if you have multiple repos. Eventually you have to roll forward your dependencies, and doing the work that goes along with that.

That's only how it works at Google if the "they" is sufficiently organizationally close that you actually care if you break them (and you'll only even know about it if you're close enough that you run their presubmit)

No, you don't have to "care". Presubmit _won't let you_ break them. That's the whole point of presubmit - to make sure tip of tree is green 100% of the time. You can't submit something that breaks the build _or_ breaks the tests. You can work with the code owner to make sure you can commit in the first place, but you don't get to just push your changes through and have the code owner deal with the fallout. Facepalm.

So is maintaining the greenness of a fragmented repo. The only difference is when you find out about the breakages.

Yes. Thank you! From my experience multi-repo doesn’t make problems go away, it just defers them and introduces others. I’m not against multi-repo or a strong proponent of monorepos... each have their strengths and drawbacks and there is a time and place for either. I struggle to deal with folks who firmly lie in one camp or the other.

Agreed!

Multi-repo makes integrations harder, because you find out about breakages later -- which means that it might be harder to identify the cause and more likely that someone depends on the behavior that broke you by the time you notice it.

But, multi-repo makes local development faster and cheaper -- you're insulated from the churn of everybody else's check-ins. You don't have to constantly refactor everything every time some dependency makes a minor tweak.

You get more done and have a smoother development lifecycle -- but you're going to keep falling behind your dependencies unless you invest in keeping up -- and that part of the process is more unfun the less often you do it.

I currently live in a monorepo. I don't love it. But I've also not loved the multi-repos I've lived in, so... shrug

Previous discussion:

https://news.ycombinator.com/item?id=11991479 https://news.ycombinator.com/item?id=15889148 https://news.ycombinator.com/item?id=17605371

Also: https://news.ycombinator.com/item?id=19303082 -- a survey of Google engineers on what they like and dislike about this.

Monorepos are cool but I hate perforce with a passion. How can we use git?

A lot of work is going into making git scale better which is nice but there will always be small open source repos. You could mirror them but it seems like it would be nice to not have to do that.

Is there any research going into figuring out how to keep many repos in sync with a single commit? Maybe you could use submodules across your company...have a single master repo that essentially just keeps track of working version sets. That might work but it sounds torturous. Is anything on the horizon to solve this problem with git?

Why do you dislike perforce? I’ve used them all and git is the worst, in my opinion, of the ones that aren’t dead and buried (like cvs). Perforce is my fave.

Internal research at google suggested that ‘git5’ users were objectively less productive than average, but there’s some friction around ‘git5’ so that doesn’t necessarily implicate git itself.

I mostly dislike the general bugginess, the need to use plugins of varying quality in all your tools, the centralized design and/or poor dvcs support, and the strange feature dead zones you get. As far as I can tell (and I opened a ticket but maybe I pulled a bad support tech) it's impossible to see what CLs were merged into a branch, only what still needs to be merged.

The general low quality of a lot of the Perforce tools has me scratching my head every day. How hard is it to make a log window that scrolls to the bottom? The p4proxy doesn't have a max size, you need to manually delete files on the machine to free space!?

You can definitely inspect the changes that have been integrated into a perforce branch with filelog and changes, it's not worth pointing out exactly how, you can find examples online (much like git). And if you don't like how p4proxy behaves when it runs out of space, just wait until you experience git with a full disk.

Git works better. You can prune git or delete lfs files remotely. P4 proxy stores every file revision it touches and does no book keeping. Revision count is not honored and obliterate doesn't clean it up. P4 proxy will grow unbounded and take up much more space than your master repo. The only way to clean it up is manually deleting raw files.

As for a single command to see already merged/copied CLs between two streams, please enlighten me. I want the unsquashed CLs. It would be really useful for generating patch notes but because p4 does version tracking at the file level and you can merge some files from a commit and not others etc, it's not trivial at all.

You have to be bluffing. I never heard of such internal research. How is even productivity measured in a context like this?

Google has an entire engineering productivity research team. They do research on tooling and such all the time.

I wasn’t doubting on existence of a research team like that. I was doubting on the research mentioned. You cannot just say people who use a particular tool (like git or git5) are less productive. That’s just absurd.

I don't think the research is public, because it's all entirely about internal tools, but yes the conclusion was that the got interface for perforce led to less efficient development across basically any metric you could look at (LoC, commit count, self-reported efficacy, etc.). Note that if you judge developers based on commit count is a bad metric, but comparing LoC throughout of developers who are evaluated on other metrics isn't problematic.

I just searched in moma with variations of “git5 productivity” and no such study showed up. Neither LoC nor commit count is a good metric to represent a developer’s productivity.

In my personal opinion, using a git/hg like interface makes it lot easier to work on a complicated CL because you can maintain internal local branches and you can easily revert your incremental changes. That’s not at all possible in perforce. I just can’t see how git/hg interface can make anyone less productive.

> Neither LoC nor commit count is a good metric to represent a developer’s productivity.

They're not a good metric to judge a developer on. But when you can do a large controlled study (or even before/after with the same developer), without the developer knowing they're being watched, it's a good metric.

Git is great (I love it) but Google anyway uses a fairly modded form of Perforce and the data model etc. is much simpler. Especially with use of IDEs it makes it so that you don't have to keep thinking about your VCS, unlike git, where GUIs come in the way.

Ah sorry. I don't want to speak to what Google has made. I've never used it. Compared to the public version of Perforce, git has a much simpler model. Streams, view specs, branch mappings, p4admin tooling, a myriad of Perforce plugins of varying quality you're expected to use, and an official gui that is embarrassingly bad make Perforce much more complex in my mind. (Although it's doing more things so it's not without reason...)

The one thing p4 does have is file level versioning. It's trivial to go back and pick single versions of files. Artists need that feature and most git clients hide it away or don't support it at all.

Google has published the git-repo tool[1][2], which is helpful for working across multiple git repos. Even with the tool there are significant pain points, though.

In the longer term, I'm hopeful that VFS for Git[3] (from Microsoft) and/or Mononoke[4] (from Facebook) will reach a level of maturity that makes it easy to host large monorepos using open-source distributed version control systems.

In the short-term most smaller organizations aren't going to hit the technical scaling limits of Git anyway. Most organizations could put all their projects in a single Git repo and not have any significant performance issues. Organizational issues like commit ACLs and code review enforcement are a separate issue, and often require additional tooling.

[1] https://gerrit.googlesource.com/git-repo/

[2] https://source.android.com/setup/develop/repo

[3] https://vfsforgit.org/

[4] https://github.com/facebookexperimental/mononoke

The short version is that you can't do this with git, because it doesn't scale.

See Facebook's discussion with git devs here: https://news.ycombinator.com/item?id=3548824

The long version is that many are trying. MS is making some headway last I checked.

Posting these because I don't see them mentioned so often in the mono/polyrepo debate.

Choosing a monorepo? Be prepared for:

- Investing in the build system

- Investing in CI/CD

- More painful upstream dependency management

- A serious investment in architecting modules and thoughtful dependencies

- Issues because of a bad deployment sequence

- Slow version control

Polyrepo your thing?

- Managing your own artifact repo (pypi, artifactory, etc)

- Painful cross-project changes

- Repeated effort around builds

Naive question

Can't we call Github a monorepository, as it is a single area one can use to access code?

Implementation details likely differ from whatever Google and her engineers are cooking up, but at the end of the day I point to an archive and pull artifacts. Certainly not every Googler clones the entire monorepo...

GitHub isn't a monorepo because you can't change multiple projects with a single commit. In a monorepo you can do things like change a function signature and change all of its callers in one commit, without breaking anyone's build. If the implementation and its callers live in different projects within GitHub, then you can't do this safely, and you need to do more work to provide backward- and forward-compatibility in your changes.

This is very important. When you're developing a large monorepo like Google3 you frequently get large scale changes from engineers outside your team, doing exactly these kinds of large scale reactors. It puts the onus of fixing things on the people making changes in the widely used libraries rather than separately on all the individual library users, which is great. It's way more efficient to have one person who knows exactly what's changing go fix it everywhere than require dozens or more separate engineers to have to investigate some new incompatibility and then potentially resolve it in sub-optimal ways.

In your example (change a function signature and change all of its callers in one commit), would it require code reviews and approvals from every affected projects?

Yes, although if you can prove convincingly that it's a 100% safe change you can often get approval from someone who has approval power for most of the repository.

In practice, if you're changing 100+ individual callsites, then you would probably make a backwards-compatible change, then use an automated system to send out and manage a bunch of different commits to clean up call sites, then clean up your old function signature once all the commits are submitted. If your code isn't a really widely used library then you probably have fewer than 20 callsites, though, so it's nice to be able to do it in one commit.

Can’t branch or monotonically commit across repos, so I’d say not really.

I’m not sure I understand what branching monotonically means, could you explain that? I’m not disagreeing with you - but is this something that can be done with googles monorepo? And when would I want to do so?

Ha, sorry!

I meant to say "atomically commit", but my brain mixed it with mono-repo, to come up with monotonically, which is a math word w/ no relevance to branching (afaik!)

For branching, I just meant there's no easy way to branch everything at once. If you want to have "weirdExperiment" branch that affects code in 4 repos, you have to go branch 4 different times. For committing, same deal. You can't easily tag a single snapshot of the state of all projects. Etc.

In a monorepo, it’s generally possible to have one version identifier that describes the state of all the code in it.

Additionally, you generally get more flexibility of how to slice and dice your view of the repo, rather than being locked into code boundaries that were set once and are difficult to change.

Is a monorepo more than just semantics?

At that scale I need tools to manage the monorepo. What about making tools to manage many repos together?

You should take a look at: https://github.com/mateodelnorte/meta

Ericsson has another philosophy where there connect multiple repos at ci/cd-time. They have open sourced a tool for that called Eiffel [1]. There is also a book [2] written by the author of Eiffel that is quite good. One of his argument is that when, as an enterprise, you buy a company with a big mature code base, you can’t just move it into another common repo with all custom tooling (also very anti agile to force everyone into the same suit). A big difference though might be that Ericsson deals with a lot of custom hardware for telecom networks. So their ci tooling might be more complex than google’s. Also, continuous deployment is not really an option for them. Then it is better to just have each piece sent out events on what’s happening (builds, test runs etc) and let event listeners in other parts of the ci/cd pipeline work out what to do.

(I have worked for Ericsson previously for 7 years but that was before Eiffel)

[1] https://eiffel-community.github.io/

[2] https://www.amazon.com/Continuous-Practices-Strategic-Accele...

One gigantic Perforce repo that achieved such Brobdingagian scale, Google had to roll their own ultra-giga Perforce that properly belongs in a hellish mondo alternate dimension of largeness. “But it’s searchable”, Google says. Yes. By the devil, it is.

Is writing/improving tools to fit your needs considered bad? If Google needed to store an exabyte of data into MySQL and they rewrote something with a MySQL compatible APIb (PingCap) would this have been bad?

I think the sentiment is that would be solving the wrong problem.

Why do they need to store so much data in a single DB, and could they use a different approach? Rewriting tools that already work well is a big of an extreme solution, but Google also has the resources to do something like that.

> Why do they need to store so much data in a single DB, and could they use a different approach?

Well, we have examples of the opposite (Amazon uses a multi-repo approach) and my understanding is that it works fine, until it doesn't, when its awful. This often happens when there are security vulnerabilities, and Amazon is forced to, under time pressure, walk their entire dependency graph and pin all dependencies to newer versions, fixing forward as they go.

At the point where you have a lot of code, you only really have two options.

> Why do they need to store so much data in a single DB, and could they use a different approach? Rewriting tools that already work well is a big of an extreme solution, but Google also has the resources to do something like that.

I see the choices as: a) rewrite one core tool infrastructure component or b) rewrite everything - currently and soon to be - built on that infrastructure.

It's also a question of level of abstraction. Why does my application care about database sharing, replication, leaders, followers, etc. It should only care about connecting & querying. The complications of running a system at scale should be hidden as well as possible from the code you've written.

From a monorepo perspective: why do I care where my code is stored? As long as I can control visibility, hide build complexity, and expose an API to other pieces of code I have all the functionality I need. Why should the nitty gritty of a package manager, VCSs, or other multi-repo concepts influence how I write my code?

(Brobdingnagian)

Edit: or maybe Brobdingragian, as Swift said the publishers actually misspelled it.

If you're a type 0 company [1] and you have a billion lines of code, much less multiple billions, you're most likely sitting on massive amounts of code bloat.

[1] https://en.wikipedia.org/wiki/Kardashev_scale

How do they avoided leaks? - surely having a giant mono repo poses an access risk

I don't remember where I read or heard this, but if you try to download all the source code at once, they will probably block your account

I've always been curious about the general directory structure they use for this. Does anyone have insight?

Top level directories by product area, then team/project based sub directories most of which would be their own repositories if you didn't do a monorepo. There are more exceptions to this than can fit in a post, but that is the general structure.

More automated commits than human?

I can only imagine how long ‘git status’ takes to complete

Given that the monorepo is in Perforce, not git, I imagine `git status` returns pretty quickly.

The equivalent is really fast. Perforce and derivates have more in common with SVN rather than Git, so if your work is filesystem based and per-folder.

Is it really “billions”?

Yes.

Yeah. I think it's even billions at HEAD too, not some metric about total changes.

Is this strictly code written at Google or dependency loc included as well?

The vast majority is not brought in from outside and was written in-house.

Don't underestimate how many of these changes are entirely automated. There's a lot of versioned data and configuration stored in the monorepo, not just human-authored code commits.

Billions is talking about LOC. As for automated changes, yes there are lots of them but from the pov of managing a VCS they are as 'legit' as manual changes.

A surprising amount of the code is autogenerated, too. Protobuf stubs, @AutoValue classes, compile-time dependency injection, etc.

The first “big” project I worked on was a monorepo, and it ended up somewhere in the neighborhood of 300k loc/sloc, I can no longer recall which, but over a third of that was generated code.

This was a datapoint on a theory I have that projects where all the code is being actively stewarded (instead of abandoned code nobody understands) have less than about 15k (human) lines per developer. If Google’s ratio of generated code is near that, then with 40k developers they’re not far from that ratio.

we're going to make this scale even more

“It was a Perforce accident that didn’t scale and later got retrofitted with mythology about monorepo designs.”

Reply on news.ycombinator.com