Hacker News Reader

Comment by dekhn | Hacker News Reader

I was the maintainer of a third-party library used by thousands of dependent applications at Google. I have to admit, I still have not seen on the outside a system that allows me to change the version of numpy, and know that thousands of dependent applications either work or break, within an hour of making my change.

Being able to write and use a mapreduce with a high level of confidence that my code would continue to work years later was another nice benefit. MRs I wrote in the first year at Google still compiled and ran with minimal changes almost 10 years later(!) which is amazing given the amount of environmental change that occurred.

That said, somebody could still halt development across the company by changing and checking in a core file (like proto defs for websearch) without testing.

Whatever social system led to google3/borg and the amazing productivity associated with it, it was a special moment that hasn't been replicated many times.

View on HN

Are you allowed to share if the internal repository a custom version control system or is it one of the open source ones?

It is, like most of the tools in Google/Facebook/..., a custom solution. These companies need their custom solution because of their scale.

It begs the question : are big companies like Atlassian, GitHub or say VolksWagen required, by their scale, to have a slick solution in order to perform on the international market?

it's already been published. Google used Perforce for at least a decade and then cloned the perforce wire protocol when it was clear perforce wouldn't keep scaling (there's a great presentation online about google using RAMSANs to store the perforce index), but backed the repo in something like bigtable or spanner.

I'll give google credit for one thing: it can change backends a lot without too much user visible pain.

>I'll give google credit for one thing: it can change backends a lot without too much user visible pain.

Amusingly, this comment may explain how someone in this thread can be incredulous about the idea that things get rewritten every 2-3 years (which yes is an exaggeration).

Google is very good at making sweeping infrastructural changes (generally improvements, I might add) without significant user pain.

People who like multirepos are always saying how easy it is to pin dependencies but like you I haven’t seen anyone doing it right since I left Google. The monorepo third-party system works well in practice.

Ps thanks for getting scipy into third_party all those years ago.

What’s really strange is how many people in that recent monorepo discussion were advocating for it in the context of a small org. I’ve only encountered monorepo in a large org, where I’m pretty convinced it’s more efficient than the alternative.

> The monorepo third-party system works well in practice.

It's worth noting that this is only viable at Google because they don't use git. Git's insistence on every client having a full copy of all history of every file in the repository makes monorepo much more expensive.

I see conflicting reports over whether google use Perforce or something proprietary called "piper"?

AFAIK they used perforce in the past and then built their own later.

This paper has details and history regarding Perforce and Piper: https://ai.google/research/pubs/pub45424

Afaik it's a custom system with download-and-cache-on-demand.

Note that there's nothing forbidding you from writing a virtual git filesystem that fetches objects from some centralized repo as files are open()ed. Git on cloud steroids.

Microsoft actually had done it and pushed the whole Windows repo into git.

https://github.com/Microsoft/VFSForGit

FWIW, there is [used to be?] a git interface for the centralized monorepo. Really handy for managing multiple dev braches at the same time.

Vfsforgit let's you do that

> Git's insistence on every client having a full copy of all history of every file in the repository makes monorepo much more expensive.

You might be happy to know Microsoft created a Virtual File System for Git[1] [2] so you do not have to have every file checked out in your working directory. Microsoft uses Git in a monorepo (for Windows, and it's 2.5 million files/300GB[2])

1. https://github.com/Microsoft/VFSForGit

2. https://vfsforgit.org/

I used a git wrapper for google3 repo. It wasn't great. There are a number of semantic differences between piper/perforce and git that made it awkward. Especially code review- git doesn't handle code review well (I still find this to be an issue with github and other sites that have code review). but it was not an officially supported solution and I believe the replacement for it is based on another DVCS, Mercurial, for some silly software engineering reasons I don't like.

I used the git wrapper for a few days until I hit a day-ending `git gc`. That's when I knew git was terrible.

They used to use Perforce and then actually outgrew it (!), implementing an initially backwards-compatible backend called Piper.

(It may still be backwards-compatible, but it's been years since they turned off the last real Perforce and I haven't worked there for years myself. So it may have diverged.)

Pinning dependencies is easy, and gets one stuck on Java 1.3 and IE 6. Since the library maintainer doesn't know it broke dependent code, the dependent code will be unlikely to have a smooth upgrade path. It's just a matter of having the luxury of picking up a time to pay the cost of dependency upgrades. Which cost may be high in a world where your dependencies also pin dependencies, likely at different versions.

The underlying assumption it that the project will fold in 2 years anyways, so one may get away with never doing dependency upgrade.

That’s a good point. Do we have any solution from companies who use git/multirepos for their libraries? Is there anyway to compile refactored libraries against existing code?

For various public package managers, there's tools that automatically send e.g. a GitHub PR for an updated dependency file when a dependency updates, which then is tested by CI. I guess something like that could be made for private repos and test versions of packages too, reporting back to the authors of the dependency.

And for anyone that wants to learn more about how Google manages third-party code, all of our docs are public at https://opensource.google.com/docs/thirdparty/

I think the problem is more just a reversal of perspective. It’s up to each different app to detect and respond to a breaking change from a dependency.

The idea of wanting to make a change to a third party library and then see all the downstream consumers who would be broken by that change if they updated to start consuming that change is an incredibly stupid thing to want, and it’s no measure of success whatsoever to build something that gives you that information.

It’s like the most giant case of coupling you can imagine (letting the statuses of thousands of consumer apps act as any type of constraint on the developer choices of the third party library, as opposed to all those consumer apps opting in to changes on their own terms by updating their dependencies).

Imagine if I have shared a bunch of copies of my resume with a bunch of recruiters. They are out there selling me as a candidate or whatever. Now I decide I want to change my resume, but I don’t know if it’s going to upset the approach some recruiter is taking.

If I can’t update my resume unless I first consult a big oracle that tells me which recruiters will be negatively impacted, that’s a problem, and not at all some type of live-with-able “customer service” positive thing. It’s just plain old bad coupling.

Creating such a system that could automatically diff the old resume’s usage constraints against my proposed changes would be a gigantic waste of time. The exact opposite of something to celebrate.

I say this as someone who routinely writes in-house software libraries used by dozens or hundreds of other apps, various teams, and even a few that are open source.

The primary thing gauging the health of our development is that we are decoupled from any consumers. We are free to make whatever changes we want, and whether downstream teams would like to receive those changes is wholly an opt-in process with versioned dependencies and easy rollbacks controlled by those consumers.

I agree in principle, but in practice its important to remember that your package only exists to be consumed by its dependencies. That is its sole purpose. If your changes aren't serving those consumers, then they're the wrong changes.

It's also useful information in the edge case where consumers are relying on undocumented or unintentional behavior in your package.

Yes, you don't want a hard-constraint of no-breaking changes ever, but knowing immediately when a change is breaking change (especially if you didn't intend it to be) is useful.

When you say,

> “in practice its important to remember that your package only exists to be consumed by its dependencies. That is its sole purpose. If your changes aren't serving those consumers, then they're the wrong changes.”

I agree completely and that’s exactly why you want downstream consumers to opt-in to your changes.

As the library writer / maintainer, nobody knows better than you how to implement the behaviors downstream consumers want. Sure, those other folks know what they want, but are not at all a trustworthy signal for how to solve it for them.

If you are constrained by what breakages your new approach would introduce, this is backwards, exactly from the “in practice” perspective you described. That means you are not able to actually solve your consumers’ problems, create new solutions, refactor old bad ways of working, because you are coupling the what with the how.

The fact that you are beholden to your consumers is all the more reason to decouple the development process from the delivery process. It only makes this idea of wanting a big oracle to tell you what would be broken not because of functional incorrectness but because of a consumers lack of accommodating the new changes all the more egregious.

Reply on news.ycombinator.com