How Google migrated billions of lines of code from Perforce to Piper
graphite.dev
graphite.dev
This is something that surprised me while I was at Google too. Of course a source code version control system shouldn't be a dependency of production, but it turns out it is just so convenient to go from (1) use your source code system to store configuration so that they are versioned and tracked; (2) have tools to read configuration and apply them automatically, Terraform style; (3) have tools to modify said configuration automatically instead of manually. Voila your source version control system suddenly becomes a NoSQL database.
There are tools that do similar with JSON but none are as simple and unobtrusive as protobuf tool chain.
Because it's probably buggy and not tested with commits past a certain file size.
Better question, why no use a more appropriate, instead of convenient, tool for the job?
Because ML engineers also want version control and easy history and rollbacks and pull requests with code reviews for model changes.
There is no tool that does that except for the version control systems that programmers use. Other disciplines doesn't seem to care as much about these things.
Edit: And in general the extra storage/compute cost of storing it there is less than the saved work in programmer hours from having version control. It is much less worth adding support for larger files than trying to build an entire separate version control solution. Google already has support for images in the version control system, with image diffs etc, there is no reason not to also add support for these large ML models similarly.
That's not an answer so much as an excuse.
> There is no tool that does that except for the version control systems that programmers use.
It wouldn't be much effort at all, especially for Google, to write a simple wrapper that relied on a database or zfs snapshots or anything actually more suitable.
https://www.microsoft.com/en-us/research/uploads/prodnew/202...
We did abuse piper for all sorts of things though, most notably “golden” screenshots for diffing ui code changes across multiple search results pages.
I would love to hear the reasoning behind going custom, and if that ended up truly being worth the investment.
Or maybe I'm misremembering the timeline on when certain things became available. shrug
It’s a lot easier to shit on a big corp. Intellectually lazy, too.
(edit: the point being that at least one of those tools existed when they started Piper if I have the timeline correct).
On a single project? Verry much doubt either of them do
The point is that either product should theoretically have the ability to handle Google's load. They were both built for scale. If I have the timeline correct, they existed at that point. I would expect those organizations to have positively salivated at the possibility of having Google as a customer; they would have bent over backwards to make it support Google's workflows.
The ROI of such a choice vs going custom is the interesting question.
Of course, if I have the timeline wrong, then the discussion is moot.
Lots of tiny repos like GitHub? Easy sharding problem. They don't even need Paxos. One large monorepo? Everything changes. Let us not forget that GitHub cannot even handle the Homebrew repo be cloned and updated by Mac users and asked them to switch to a CDN.
this is the perfect HN comment - so confident but with so little thought.
essentially all google code - something like a billion LOC - are in one repository.
there's something like 100 000 people who use source control, plus jillions of systems doing automatic commits all the time.
it's atomic on each commit.
so that's several QPS globally synchronising a huge data store, with the entire dev tooling system hanging off it.
git is ... very very much not suitable for such systems.
gitlab is just a simple web app reading from a bunch of small and rarely changing git repositories.
Piper is also the successor to Perforce, which is a completely different model to git.
How repos are organized is a choice. Google could just as easily (financially speaking) have chosen to break up that repo and use more standard tooling. They did not choose that. Did they have (possibly perfectly valid) reasons for that? Sure.
That does not render it as their only possible path.
Rather than switch their source control strategy to something that would work well with existing products, they chose to build a brand new product to support their existing strategy. This is, IMO, not an obviously sane decision from an outside perspective. Why?
Because source control has nothing to do with their core business, and in addition to the cost of developing and deploying that solution, they now have to bear the cost of maintaining it, which is likely not insignificant.
That's how it looks from an outside perspective.
The question I'm asking is not "why didn't they use git?!". The question I'm asking is, "what is the ROI of building and maintaining their own solution versus biting the bullet and pivoting so they can use pre-existing products?"
Haven't seen an answer to that yet. People are too busy implying or outright stating that I'm stupid for asking the question in the first place.
not at all, it's just what google decided to do on balance. not only did they decide it was the best choice for them, the cost of switching would be ... astronomical. the amount of tooling that assumes it can find all code in the one place is very very large.
A single senior developer full time @ Google is probably a half million dollar a year investment, give or take. How many do they have maintaining it? How long before that yearly cost exceeds both license fees for an alternate solution and the cost of development and migration?
The answer to that is not obvious.
(edit: response specifically regarding "astronomical" costs)
I wondered if it would have been cheaper overall to use pre-existing tools and adjust things than to build a whole new product that has nothing to do with your core business, and then have to maintain that product. This is a wholly different argument.
TANSTAAFL right back at you; development and maintenance time is not free either, and there's a potential argument that pivoting their other tooling would also open them up to other off-the-shelf opportunities, though that is much harder to evaluate.
I've worked at scale. I am well aware that sometimes custom is the only way. I've been there and done that, and walked it back when someone finally did something off the shelf that was cheaper to maintain. Custom comes with a cost, and that cost can often be hefty.
Source control... is not usually where you hear the "it should be custom" argument, and even less often is it at all supportable when you look at the big picture.
Google is certainly large enough to have unique requirements, but that alone doesn't justify the choice.
I thought about the larger organizations, and of them I could identify only Microsoft and IBM as other companies who have their own SCM products (though that's probably ignorance; there are probably more ;). The difference with them, though, is in the fact that they sell those products as products, making them part of the core business. Profit center instead of cost center.
Google doesn't do that AFAICT, so all that investment in development and maintenance is nothing more than cost at the bottom line. And not just monetary cost: what else could those developers be doing with their time?
Hence, it really makes me wonder if there's a bunch of developer religion that's costing Google a lot of unnecessary cash. If I were by some strange miracle brought in as a CTO at Google? I'd be wanting some pretty hard facts on that (or a plan to productize it) to justify its continued existence.
If someone tells me that monorepo (or multirepo) is better, I'll respect their opinion even if I disagree with it. If, however, someone tells me that one or the other is the only way, and they must have a custom source control system to prevent things from collapsing like a house of cards?
Yeah, I'm going to call bullshit.
Maybe that is the right decision. But Google has organically grown over the decades and their tooling evolved with that.
It's hard to say which requirements they were working with and what they were trying to optimize for - sometimes higher costs are just okay if they allow or enable other things.
For me it's always "what are we trading?". In an organization the size of Google, it's far easier to amortize away the cost of such a thing. That doesn't mean it's not a cost, though. And there's a lot of tooling around modern SCM that they won't get because they're on a bespoke system, so there's a cost there too.
As you say, at the time the variables may have been very different.
It's just an area where if I had to make that decision (even back then), someone would have had to be exceptionally persuasive to convince me to authorize what is effectively a multi-million dollar project... to recreate source control. =)
Using made-up numbers, if you have 50000 employees using source control at your company and a custom solution would improve their productivity by 0.1%, having a team of 10 building such a solution is positive ROI. In contrast, if you have 1000 employees, it would be a negative ROI. This is why it's worth doing if you're Google or Facebook but not most other companies.
Perhaps Google tried to do it this way and hit the same issues.
"The engineers tried running a simulation, creating a dummy repo that matched the expected scale of Facebook’s codebase in a few years. The result was horrifying - basic Git commands took over 45 minutes to complete."[1]
> Because source control has nothing to do with their core business, and in addition to the cost of developing and deploying that solution, they now have to bear the cost of maintaining it, which is likely not insignificant.
God I hate this outsourcing mindset. It's so refreshing to see a company value their developer productivity that they make a custom tool for just their needs. The ROI is readily measurable as the increased productivity of hundreds of thousands of people writing code. It's also measurable in the simplified tooling of every single tool that needs to interact with the source control. Same thing with the decision to switch to multi-repo from a monorepo: lots of added overhead to every engineer and every tool.
Git sucks so much compared to Piper that it boggles the mind to even ask such a question.
I don't care if the competing solution would have been git, Perforce, hg, or whatever. The point was not "git". The point was "off the shelf". I also do not dispute that Piper may well be a great system; I've never even seen it, so I wouldn't know.
The issue is not about "outsourcing" (which is a rather loaded term that should not be substituted for "using off the shelf products"). It's not about "engineer productivity", per se. The question is about overall business efficiency.
I'm sure it could make engineers that much more productive to have the latest and greatest laptop every year. That doesn't mean we replace all our hardware every year. We also don't give every developer a terabyte of RAM. It's the same basic issue.
It's always a trade; those dollars can potentially do more good somewhere else.
You'll note that I have at no point stated outright that I thought they made the wrong decision; only that I'm skeptical. I asked the question about the ROI, and people decided to get all heated about it. The only arguably inflammatory point in my comment was about Google's case of NIH.
It's really hard to deny that Google has acute NIH.
Sometimes that's a good thing; they've done some amazing things and given us some amazing technologies. Other times, I question their decisions.
I don't think there's anything wrong with that.
My perspective is simple: inventing its own source control system at Google is absolutely worthwhile and the ROI is highly positive. The tradeoff is worthwhile. You might say that before Piper existed it might be a gamble whether it would be worth it. But in hindsight, absolutely 100% yes it's worthwhile. It does not need to be argued, for anyone who has used it.
The only thing that is really subject to debate is whether Google's engineering leads had the foresight of what Piper would accomplish or whether they simply wanted to build something custom due to NIH and it only fortuitously turned out to be good. That's a question no one here can answer.
Bah. Text boxes in browsers suck, and mine's not cooperating at the moment, so instead of writing another book, I'll just say: thank you for the reasonable response. =)
But it is easy to think of a company that is of Google's scale failing to complete such a migration, wasting years with many projects not being able to migrate and then ultimately having two systems that everyone has to sync between. That they actually succeeded at doing it instead of failing is the impressive part.
That is such a cringy, fearful, statement...
Though perhaps I lack perspective, having just toured Normandy where the D Day landings took place.
Affiliated with, no. Unless the author was there, just made everything up or sourced the details via 3rd party reporting (or as is the case in the opening line - via google published papers/press releases) then yes google PR personel would have been involved.