Hacker News Reader

Comment by bushbaba | Hacker News Reader

Google’s CI/CD and internal monorepo ecosystem building against latest have a high cost to maintain legacy code. You can’t just let something rot with minimal maitenance at google.

While fine for a customer tech ecosystem facing security threats, it’s really not ideal for enterprise offerings.

View on HN

That's a weird take. Is that your first-hand experience? My personal experience was that google3 code can be on autopilot for decades and nobody will break it, because the testing system keeps people from breaking it, and the large-scale changes infrastructure won't break any tests.

I think you missed the point. The cost is how hard it is to make changes because of not breaking all those tests on legacy code.

There's only one version of each library allowed for essentially the whole company, including internal libraries. So if someone wants to upgrade it they may need to update the old codebase's call sites and get those changes approved.

My impression is that the hassle and pain of doing this is often why languishing projects are deprecated. I can't tell you how many times I've read TotT issues or seen threads on eng-misc or elsewhere that would either talking about some new test system and how it handles crufty codebases, or how one should check for dependencies, or what might happen with some internal app that lost its sponsor/maintainer because they either transferred teams or left the company. For a while, even somewhat important things like Dory were un-maintained (after Moh Musa left Google), and lots of times it's happened that reorgs caused deprecations.

Seeing how those sorts of things play out in actuality, it doesn't surprise me that unpopular or unprofitable products are killed rather than kept on life support. Lots of times, the people who would have to fight for them don't have the political will (or maybe even complete history of the product) to commit to staff allocation to keep them alive in the face of the alternative: chasing the next 10x dream.

The weight of managing depot wide upgrades are definitely not a factor in turning down customer facing services. In 10+ years doing low level libraries and infrastructure no one ever used my teams efforts as a reason to sunset an offering.

Although depot migrations are tedious, the burden of doing them for all but a subset of mothballed systems would not significantly changed my life.

Also, generally speaking, because the model is "library owner makes the LSCs," library consumers are happy to accept small/reasonable changes in API usage.

Off the top of my head I can only think of one time when an internal library deprecation caused a huge ruckus: some front-end framework was deprecated which led to a huge push to rewrite many of Google's internal websites.

When I left even BigTable, MapReduce and other services are still supported even though they have been deprecated and their replacements (Spanner, Flume) had been GA for more than a decade. (Although their API calls may just go through a translation layer, not sure.)

No the real reason is Google's internal systems do not depend at all on these services. Therefore no one values the maintenance effort required to keep the lights on.

If anything the CI/CD infra backing Google development keeps these services running longer than they would otherwise.

So the same is effectively true at Amazon? If a service in a version set isn't actively maintained then it'll eventually be unable to merge from live, and that means its dead code. Version sets do allow for services to exist in a degraded state (ex. can upgrade some dependencies, but not others) but without consistent maintenance services will fall off a cliff.

if you're referring to Z builds in version sets, where you build a specific package but not it's consumers, those have not been allowed for several years now.

if you just mean "you don't have to merge everything from live, you can just merge select packages, or build specific updates in manually" then yes that is true, but you won't have a fun time doing it for very long.

i mean the latter (have z builds been allowed in the past decade), and i agree that it’s not fun over the long term but possible and in a lot cases sustainable.

Z-builds were possible until BEARS launched in 2018-2019ish

Monorepos are garbage and Amazon outcompetes Google by not having one.

Everyone in the Valley thinks they are awesome because Google does it and a Stanford grad is obsessed with cross project atomic commits.

Keep going and Amazon is gong to bury Google cloud permanently.

You think about product lifecycles differently when you have a monorepo. Breaking change in a library? Just fix all the breaks in the same commit. Dependency hell? Not a problem when you have an ancient version forked in your monorepo. These are real problems customers face when they're not in your ecosystem, and Google doesn't always realize that. It was this same for App Engine. It wasn't a model 95% of customers wanted, but for Google spinning up services internally, it was great.

That’s fair, I view Google as internally socialist, which I think works great until it hits a knee and doesn’t.

My experience with Facebook was that Mercurial was awesome, but the multiple monorepos were unbearably slow.

One of the great ironies of capitalism is ~all companies are socialist internally, or at the very least command economies.

Wouldn't that make it easier to maintain? Don't need to task specific headcount towards it for general maintenance.

Then maybe the monorepo model doesn't work for them....

Reply on news.ycombinator.com