Ask HN: What is the development environment at Google like? | Hacker News Reader

Ask HN: What is the development environment at Google like?

33 pointsralmeida9 comments

I take a lot of interest in the setup of development environments and how they affect productivity, and I hear a lot about the development environment at Google specifically (mainly on Quora and 'word on the street').

In particular, it seems that for most work, the actual machine being worked on is just a 'terminal' for a more powerful machine accessed via SSH where the actual work (testing, builds, etc, is done).

I remember having read somewhere that developers are even not allowed to check out the repository in their local machines.

Is all this mostly true? Does it apply for all types of development (backend, frontend, mobile, tooling, etc)? I can see this working for backend development, but for mobile (or even maybe frontend), it sounds like a big hassle.

I would be interested in specific accounts of day-to-day work as well. For example, if you do use a remote machine for doing some type of work, does this mean that debugging is done accessing the IP of the remote machine directly from the local machine? Or all testing is done directly in the remote? Is this remote machine a remote desktop, or just a terminal? How is the connection between the machines made (apart from the SSH terminal the developer is operating)? Directly over the Internet? SSH tunnels? VPN?

I would also be interested in accounts from the other big players (Facebook, Amazon, Microsoft, Netflix, etc).

DISCLAIMER. I don't work there. But I know professors and met people that did/still do. It does depend on the position.

First, their infrastructure and resources are mind-boggling impressive. Second, it is more like any other job you would normally do. You have deadlines, get it done. Somethings are more fun to work on than others. Yes they have pros(including perks)/cons like any other job. Third, you loose your life to the company. They normally are at the office all day and Saturdays. I know people who love this and others that hate it. Essentially, all your coworkers become your friends because you are always with them.

Yes, I am told they have their own Ubuntu flavor they use to work on. Servers are constantly building software. Every company has a VPN, this is normal business practice.

I used to work for Google - voluntarily left earlier this year. I'm not sure how much I am allowed to disclose about internal tools and processes, so I'll refer you to this article: https://cacm.acm.org/magazines/2016/7/204032-why-google-stor... and call out some parts that you might care about.

The workstation is not connected to the codebase via SSH on a server, but rather the version control clients are remote and mapped into the local filesystem via FUSE. So the workstation is less of a dumb terminal; programs that access the code can run locally, even though the code itself is not stored locally. Therefore, version control clients can be accessed with various web-based tools as well.

Not all developers used a workflow that includes Piper/CitC, but most do. From my limited view, the Piper/CitC workflows were the most painless.

Debugging - I only ever used a debugger for Python, and in those cases the application I was running was local. Otherwise I never used a debugger but rather logging (printf debugging, essentially). I'm of the opinion that if you have to open a debugger to understand the code, you've already lost. Others might have better stories.

Remote machine - as mentioned above, there is no remote machine. Things are done on your workstation. The build commmands can build things locally (slow) or farm them out to a cluster of build machines in datacenters (fast). See https://bazel.build/faq.html#does-bazel-require-a-build-clus...

That's very interesting, thanks for your perspective. I was familiar with Piper (from having skimmed the very article you linked), but didn't go much in depth about how it impacts dev environment.

Debugging, I tend to agree with you to some extent - I'm even 'on the record' with friends on saying that debugging should 'come from the mind' before mindlessly stepping through a debugger. Being able to do it, however, may be a valuable tool to help you 'refresh your mental model' of the code, or even to build the mental model in the first place when you are first learning a new system.

Building (Bazel) - Nice! Typically, when doing development on a compiled system, would you retrieve the build back from Bazel/Blaze to 'play around' with it locally? Or is most development test-driven? If the former, how many builds did you do on a typical day? If you're able/willing to share, of course.

I have a friend that works there. He said everything he does runs in App Engine, and when they go for a feature update, they just push it off to one of several staging environments running App Engine, which is maintained by a team of devops engineers. From his perspective, it's just push a button and go.

He didn't mention anything about a remote machine.

Disclaimer: this was all information I gleaned from casual conversation, so take its accuracy with a grain of salt.

The answer to this question depends quite a lot on what is being developed. It's not like an engineer at Google could have their own local copy of google.com.

Most of my career I've been in mobile development, so the code basis are typically smaller. I've always had the full code on my local development machine. But you have to connect to web services and back end databases, so there's usually development, test, and production environments.

Interesting question indeed. E.g. myths such as everyone using either vim or emacs... I wonder how common are "sane" IDEs, e.g. someone who uses C++ using Visual C++ IDE, Windows instead of mac/Linux, etc...

I guess I'd ask, why do you care? Are you looking to optimize your own workflow, or are just curious about Google's?

I don't think we need to be holding an advertising company up as our best achievement. At least I hope not.

Fair question. I want mainly to optimize my own workflow, maybe try some new 'ways' do develop software - editing remote vs local, building remote vs local, testing remote vs local, manual "printf" debugging vs debugger, etc.

I'll tell you why I chose to focus on Google specifically to start with.

First, it has a generally "analytic approach" to running their business, which they use for many things, from actual engineering, to dev ops, to "people ops" - you might have seen an article which describes how they basically tackled the problem of "how team dynamics" work with a large-scale internal social experiment with a heavily quantitative methodology.

Second, being very large scale, they have plenty of incentive to try to optimize developer experience, since even a small efficiency gain can make a non-trivial impact.

Third, they share quite a bit of their 'opinionated approaches' about things both in the form of research, articles and open source software (see MapReduce, Piper, Kubernetes, etc), but not everything, which seems to spark the imagination of people and generate some 'folklore' (as I mentioned in the question, I've heard/seen quotes from 'no one can check out code locally', 'everyone uses vim', 'nobody uses vim', etc).

This is obviously only my impression, which may be even way off.

Finally, I am sure that other similarly sized companies may have very interesting insights about this kind of thing. I chose to not make the question too generic also to not dilute the interest of the very people qualified to offer some insight.

BS. It's like anywhere else, you code on your machine and devops takes care of the rest.

Reply on news.ycombinator.com