Hacker News Reader

Comment by stdbrouw | Hacker News Reader

> Since all code is versioned in the same repository, there is only ever one version of the truth, and no concern about independent versioning of dependencies.

This sounds like horror to me: it's essentially a forced update to the latest version of all in-house dependencies.

Interesting article though. It feels like there's a broader lesson here about not getting obsessed with what for some reason have become best practices, and really taking the time to think independently about the pros and cons of various approaches.

View on HN

It's a "forced" update, but the downstream dependencies have veto power: if their tests fail, the change gets rolled back.

Of course in practice, it depends on how important the app is. If you break gmail it's definitely getting rolled back. If it's just a one-off abandoned tool somewhere doing something they're not supposed to do anyway, or some tests that flake all the time, maybe it will be ignored.

If you are upstream of a lot of important code, you might not be able to move as quickly as you like and that's because you have a lot of responsibility.

Migrations need to be planned carefully. Unlike in open source, you cannot make an incompatible change, bump the major version number, and call it a day. Migrating the downstream dependencies is your job. Version numbers are mostly meaningless and compatibility is judged by actually compiling code and running tests.

You soon learn that if you can fix a bug without changing a public API it's a whole lot easier. But if you need to change an API, there's a way to do it.

It's been a long, long time since I worked at Google, but IIRC the build system was like Maven: you could depend on the latest version of something, or a specific version, and only the dependencies you needed would get pulled from the global repo when you checked out code. If your code depended on a specific version of some piece of code, then you'd be immune to future updates to that piece of code unless you updated your dependency config. (Hopefully I'm remembering all of this correctly..)

Nope, that's not quite how it worked. Except for a few critical libraries that were pinned to specific versions, every build built against HEAD. However, a presubmit hook would run tests on the transitive closure of code depending on the changed code, and if anything broke, it was your responsibility as the committer to fix it or roll it back.

This is really what kept things stable. If you maintained a a common library and wanted to update it, you had to either keep things backwards compatible, write an automated refactor across the codebase (which wasn't too hard because of awesome tooling), or you would get dozens of angry emails when everything broke. If you relied on a library, you had to be sure to write solid e2e tests or your code might silently break and it would mostly be your fault.

I definitely miss the big G, so many resources were dedicated to engineer productivity and tooling. It's what makes the monorepo work.

Thanks! I appreciate the correction. I miss a lot of the resources, too. I also miss the spirit of dedicating serious attention to code quality. Things like Testing Fix-it and Documentation Fix-it days were awesome.

> Things like Testing Fix-it and Documentation Fix-it days were awesome.

Can you elaborate on this? Are they just days where the team goes "Ok, no need to stress about new features today, let's just catch on on test coverage / documentation" ? If so, that sounds pretty wonderful.

It was company-wide. Every month or two, there would be a "documentation fix-it day" or something similar. The theme would vary: sometimes it was docs, sometimes unit tests, sometimes localization, etc. If memory serves correctly, the idea was that unless you had a high priority/production bug, you'd spend your day working on the theme of the fix-it.

Why did you leave?

"Oh yeah, and this is our awesome in-house build system. You're really going to enjoy working on ads!"

Yes, working in a product area that's directly responsible for generating $N*10^9 of revenue per year is just terrible. I'm sure they don't do ANYTHING interesting, technology-wise.

I'm being facetious and exaggerating a bit, sure. But for every engineer doing cool machine learning stuff, there are 10 mucking about with AdWords and AdSense front-ends -- on customer-facing stuff that they probably don't use themselves. It's energising to work on a product you use, it's draining to work on one you don't.

(And I guess I'm putting words into the mouth of the earlier poster, sorry about that. This is from my own experience, not theirs!)

Just curios how does this automatic refactoring across entire code base work? Do they build AST's and then refactor? Or it at more surface level where you are replacing text intelligently?

Either one. http://research.google.com/pubs/pub41342.html for the former.

FYI, Google open sourced their build system: http://www.bazel.io

No, it isn't if artifacts/projects declare dependencies against fixed, stable version archived somewhere (e.g. artifactory).

Just I wonder the repository performance and the every day workflow.

Pretty much everything builds against head, and very little is in versioned archives.

There's a ton of fantastic artifact caching (that gets invalidated whenever source changes; which is frequently), distributing building, and some really nifty source code juggling

Mmm, two things of note: 1) As has been mentioned, I'm fairly sure you can fix on a given dependency. 2) Changes to upstream libraries trigger tests on consumers of that library, so breaking changes tend not to happen.

>Changes to upstream libraries trigger tests on consumers of that library, so breaking changes tend not to happen.

This doesn't prevent build breaking changes, it just makes the team using the library aware what is broken and they can have fix ready when they want to upgrade their dependencies.

Afaik you cannot submit code that breaks any test in the repo. I mean, there are ways to force it, but unless it's a special case you really don't want to.

Judging by the downvotes, obviously some people have no idea how maintaining libraries work.

I think the way you should think of it is that there is only one "canonical" version - otherwise the "canonical" version is fragmented, and the fragments can contradict eachother.

And god damn, I hate Mark Shuttleworth for using that name for his company.

Reply on news.ycombinator.com