Hacker News Reader

Comment by ChuckMcM | Hacker News Reader

I will say that I saw and experienced many things that changed my definition of 'large' at Google, but the most amazing was the source code control / code review / build system that kept it all together.

The bad news was that it allowed people to say "I've just changed the API to <x> to support the <y> initiative, code released after this commit will need to be updated." and have that effect hundreds of projects, but at the same time, the project teams could do the adaptation very quickly and adapt. With the orb on their desk telling them at that their integration and unit tests were passing.

I thought to myself, if there is ever a distributed world wide operating system / environment, it is going to look something like that.

View on HN

Can you (or somebody) explain how that works in practice?

Here I am ready to deploy some new feature to gmail. In the meantime, I'm getting a steady stream of API changes. Can I build and release gmail to a specific revision number, and only incorporate the changes when I am ready, or are all release essentially off of the tip of the tree.

I don't need specifics, just the general idea. Where we work basically every project lives in its own branch, which makes it essentially impossible to synchonize changes. Things have to get merged to trunk, then pulled into the appropriate branch, and I don't like it at all.

Do you have version numbers for libraries/components: projext X uses version 1.5.4 of Y, 32.4.18 of Z, and so on, do you pull by revision number, are you all on tip?

It'd be very interesting to hear how you manage this.

Every feature has a flag that you can turn on/off to your heart content. Releases are cut at head. No need to use the version control hammer to screw all the zippers.

Would that be analogous to, every feature is it's own git branch? and you choose which branches to merge when building?

How are conflicts managed?

No, features are configurable at runtime. This allows A/B testing or slowly rolling out a new feature. Depending on the type of change, you might test it on 1% of users, 1% of servers, etc.

Every feature is in HEAD, behind a flag. Precisely because managing conflicts cannot be automated, relying on branching is unwise. With HEAD based development, you manually solve conflicts exactly once, when submitting to HEAD, then you never have to think about them again.

Local branches for development, a single HEAD for merging code.

In theory: Everyone works from head, so there is only one version of everything.

When you write a feature, you follow these steps.

1. Write code and submit to the main (and only) branch. Hide your feature behind a flag. 2. Releases happen from head (with no regard to your feature) 3. When you want to enable your feature, you flip a flag to turn it on.

In practice: Things are messier.

Yes, you sync you workspace to a particular revision and work within that. Syncing is a manually triggered process. And generally speaking, people changing APIs are required to find and fix all users (which isn't that hard with the infrastructure available). There is still some churn created by the constant stream of changes, but it's usually not on the changing APIs level.

The solution to the excessive API change problem is to force whoever changes the API to fix all the consumers himself before the change is accepted.

The Linux kernel generally uses this policy for internal APIs for example.

We do, mostly. Because Piper is a global repository, we have systems to do global safe refactors, and do so often. If the API changes drastically, there's usually a lengthy deprecation period before the API is switched over.

I guess that works for the Linux kernel, but I would presume that for a large distributed operation like Google it would be much better to simply deprecate/version APIs and have the project teams update to a deadline.

I mean, it's presumably impossible to have a single computer running a single OS build all of the Google software and run the testing.

Google is rumored to have an extremely powerful distributed compilation farm. It wouldn't surprise me if a single developer could make a change that affects the entire codebase and test it themselves.

The key to this in her talk was TAP

  > The solution to the excessive API change problem is to force
  > whoever changes the API to fix all the consumers himself
  > before the change is accepted.

This doesn't seem scalable. Let's consider the case of one api endpoint being changed by one developer, to add a new param to a function call. Further assume that this impacts hundreds of projects.

Does it really make sense to make one developer update those hundred projects? Not only will it take forever for it to get finished (possibly never if there are new consumers of this api coming online frequently), but the developer of the core api may not have any experience in the impacted consumers of this codebase. I think the end result of this policy would be nothing once written ever would get updated, and new apis would just be added all the time (api explosion).

It's scalable with the right tools. If you can write a transform at the level of the AST that will make the change for you, you can do it in one commit. FB has written about this: https://medium.com/@cpojer/effective-javascript-codemods-5a6...

Not that it's a silver bullet, but it can make a lot of these cases non-issues.

Google's product explosions / surprise deprecations possibly hint that this is what happens? Changing the API becomes cumbersome, so you just make a new product with a new API to do an end-run around the requirement...

If all your RPCs take a single parameter, this isn't a problem provided you use universal defaults.

It seems bad but the alternative (wait forever for groups that don't understand the change and may not even exist anymore to update their code) seems worse.

It maybe isn't scalable, but that's part of the benefit. If you want to make a change to a widely-used API, it's going to be a lot of work, and it's not going to be a lot of work for the users of the API, it's going to be a lot of work for _you_ because _you_ are required to do it yourself. This prevents a lot of API churn unless the benefit is clear and sufficiently large.

If it was any other way you'd rapidly reach a useless equilibrium where random engineers were demanding that thousands of other engineers fulfill unfunded mandates for what might turn out to be negligible benefits.

That's one extreme. Another extreme is that you have the API versioning from hell, where you can never get rid of technical debt because any and all API changes will break someone, somewhere, who has no reason to migrate, so you're left keeping ancient code on life support indefinitely.

This makes it harder to change APIs. Which has advantages and disadvantages.

> The solution to the excessive API change problem is to force whoever changes the API to fix all the consumers himself before the change is accepted.

Having people unaware of a project's purpose making changes to its code sounds like a nightmare to me.

When that's the cultural norm, people adjust accordingly. Tools include liberal use of assertions, defensive tests, and most important, code reviews that catch and remove quirkiness.

It's a nice environment to work in. In addition to hastening Noogler onboarding, it also increases employee retention. If you are an expert in your project's codebase but get burned out, you can easily transfer to another project and be almost immediately productive.

Obviously, there's domain-specific knowledge that doesn't transfer easily or quickly from project to project. But that's quite different from self-inflicted code fragility; one's an asset and the other's a liability.

> With the orb on their desk telling them at that their integration and unit tests were passing.

Anymore info on this? Is it in house hardware?

IIRC, it was built on a hacked up Ambient Orb

http://ambientdevices.myshopify.com/products/stock-orb

Wow, $150? Shocked there isn't a cheaper USB version that is more easily hackable. I mean, is it really more than color-changing LEDs in a plastic sphere?

It's also a perpetual? subscription to a

   nationwide wireless network

to obtain e.g. the latest stock prices.

blink(1) is a small USB light to give you glanceable notice of anything on your computer or the internet. blink(1) makes it easy to connect data sources in the cloud or on your computer to a full-color RGB LED so you can know what’s happening

https://blink1.thingm.com/

Thanks for the link--looks awesome, wish the form factor were a bit larger and not so directly attached to the computer. I'll bet I could create an extension cable for it and rip off the housing to make the LEDs allow more light through, then put it in a translucent plastic sphere.

Started with a project that used the Ambient Orb to show test status, they were expensive which led to a contest to make the cheapest one, basically an ATTiny with a software USB hid stack tied to an RGB LED. Henner Zeller won that contest and I think maybe 5000 gOrbs were built.

I built something similar as a hack project... Closet dome light fixture turned upside down. Rip out the innards. Install RGB LED. Connect LED up to Arduino and pull information from an API. Adjust LED based on your conditions. I spent around $75, but that was years ago, probably far cheaper now.

That's actually a good question. What would a world wide environment look like. Currently we have the web, but that isn't really a world wide environment in this sense. I think that this could be a way it would look, but more of a combination of this and a way to split everything down into the parts again without breaking the system. However, I've never worked at G, so I can't know if they have a method for this

Reply on news.ycombinator.com