Hacker News Reader

I'm a Google SRE, and the first place I go when faced with a complicated problem is the whiteboard (if collaborating with others) or the legal pad (if I'm trying to figure it out myself).

If collaborating with others in a remote office, I begrudgingly make a Google doc and treat it like a whiteboard.

Most of what we do involves enormously complicated systems with lots of nodes and RPC flows between those nodes, different pathways depending on the flavor of the RPC, systems spread out in different metros around the world... it's very difficult to visualize all that complexity in your head if you've never seen it on a cocktail napkin.

For example, just last week I sketched out a diagram inspired by the famous visualization of Napoleon's campaign to Moscow (http://www.aviz.fr/wiki/uploads/Research/minard.jpg), primarily to help me wrap my head around a complicated RPC flow (at the first node, 32% of the RPCs are classified as XYZ. Each of those spawns 3 new RPCs that go here and 1 that goes over there...) When I got stuck, I called over a colleague, and he was able to immediately see, just by looking, where I was going wrong, and with a few strokes of the marker, set me straight.

I later turned it into a spreadsheet so I could use it to explain the model to others. Also, it was nice to be able to use worksheet functions to do the math. But I never would have been able to get that far that fast without starting at the whiteboard.

Former Google SRE here. Trust me, many of the problems we faced were solved on whiteboards, during chats at lunch, or via email. Most of issues we faced were of a class that simply isn't covered on basic sites like Stack Overflow.

The interview described is pretty much exactly as I remember them. SRE is actually quite difficult to get into, precisely because you need to have fairly deep knowledge on a wide range of topics. The "ways in which you were asked to solve problems" are actually the best way to determine if an application actually knows about what they will need to know.

Heck i'd be worried if your real issues were the stuff solved on stackoverflow :P not that stackoverflow is bad, but the stuff there is generally a basic/quick reference.

When is the last time you heard of a system administrator writing down problems on a whiteboard as opposed to asking a colleague or Googling the answer?

"You're a site reliability engineer at Google. Google is offline and you have to fix it, what do you do?"

"I'd Google the answer"

"Try again"

"I'd ... Bing and decide?"

both laugh

>"When is the last time you heard of a system administrator writing down problems on a whiteboard as opposed to asking a colleague or Googling the answer?"

Once you move beyond roles which are effectively operations, maintenance or been-done implementation to anything that can be called engineering you'll start to face problems for which there are no canned answers.

>"It's not the 70's any more, people rarely solve problems on whiteboards and paper."

This has nothing to do with era, it's a matter of scope and scale. When you have problem that is big / complex enough that it can't be reasonably solved by a single person a whiteboard is invaluable in laying things out and thinking them through.

>"They solve them on the computer, sometimes through knowledge and their skill-set and other times through luck and Googling."

Where do you think all that helpful information on Google, or anywhere else actually originates? When the folks at Google were in the process of engineering GFS do you think they just Googled "chunk replica placement" and hoped to luck into a StackOverflow post on HDFS from 10 years in the future?

>It's not the 70's any more, people rarely solve problems on whiteboards and paper. They solve them on the computer, sometimes through knowledge and their skill-set and other times through luck and Googling.

This isn't universally true. Anecdotally, I often find I'm much better able to think through tough problems if I step away from the keyboard and spend some time sketching out ideas on paper or on a whiteboard. I also keep the on-paper results in a notebook, which is occasionally useful to refer back to later in a project.

Sometimes just introducing some distance between you and the problem is enough to give you a key insight. That said, the interview environment is still nothing like this. There, you're under great pressure on the whiteboard, something which is probably not true in your day-to-day.

I'd like to think this is true, because I enjoy working in that manner, but I've generally found going totally offline and solving problems gives me satisfyingly worked-out-by-myself solutions to problems that... I could've solved more quickly if I'd done more reading for half that time instead.

If something really, truly has never been done before, nor even anything similar enough to be useful to me, then yes, this is the right way to work. But it's more common that someone has actually worked on something at least related (even if not quite the same), and that I could solve the problem more quickly if I looked for what they said about it first. That might take a bit of searching and reading to discover, sometimes even a few hours of it. But usually not as much time overall as re-solving it myself does... especially taking into account re-discovering all the edge cases.

My hypothesis is that many people don't realize this because they never follow up later to check if their solution was really novel, or was just lurking behind a keyword they didn't think to try. If you do that a bit and adapt your habits to miss things less often in the future, you can get better at finding and adapting existing solutions, rather than re-inventing things from scratch. But that's sometimes a bit deflating, because then you realize you weren't inventing so many new things before, either...

What you say is true, but... if it is a critical[1] task / area for you, you're better off reimplementing it anyway. Most existing work isn't so deep that concerted effort on your behalf won't improve on it - but only if you have the time to spend on it.

[1] I mean this in the sustainable competitive advantage sense

If I really need to think about a tough problem, I tend to get a pad of paper and go sit outside on my deck and think about it away from a computer. Sort of an "astrophysicists are not in the telescope business" thing :-)

Having heard numerous accounts of the Google hiring process, both successful and unsuccessful, I'm convinced that it serves a dual role. In addition to brining in qualified employees, I believe it's intent is to reject qualified candidates. Those qualified candidates will end up being engineering leaders in companies that Google interacts with and the more that they can maintain the impression that Google engineers are the best of the best, the easier those interactions will be and the better received Google products will be. It also means that when Google calls and invites someone to interview, they almost never get turned down.

>they can maintain the impression that Google engineers are the best of the best

This is not true for everyone. I see it as more of a failure on Google's behalf to create a good selection process. The flawed assumption you're making is that, since they have a noticeable false positive rate (i.e. good people getting rejected), they don't have false negatives (i.e. unqualified candidates getting offers). There is no guaranteed correlation between false negatives and false positives.

To carry this a little further, I would argue that it's very likely that some bad engineers get into Google because, by definition, their selection process is not correctly picking good engineers - just a rough approximation of what they think makes a good engineer.

I don't doubt that some bad engineers get into Google...I've met quite a few. I met one engineer who believed that you should avoid interfaces in Java because it makes it difficult to click through source code in an IDE. I've interviewed ex-Googlers who were completely lost answering the interview question, "how do you write maintainable code?" From what I've seen, Google isn't testing for being able to write maintainable code and is, instead, testing almost exclusively for problem solving and being able to apply algorithms/data structures. That, alone, is going to lead to hiring some bad engineers.

But you have to look pretty hard to find those engineers...much harder than you do to find quality engineers that have been turned down by Google. And I believe it's intentional...that those false positives are about seeding the rest of the industry with people rejected by Google. I believe they interview more candidates than they need to bring in to fill their open positions in order to feed the perception (not the reality) that Google's engineers are the best of the best. That's the perception they care about, not the perception that their interview process is good at choosing employees.

Not true. I'm not a big shot but I've turned them down. Had two rejections from them in the past .. don't want to waste my time.

If this is the actual goal they are not even bad at hiring, which I can testify personally, they are also bad to be hired at. They cannot offer interesting jobs, their hiring practice and personnel sucks, their pay is not really competitive. Who wants to work for Google if you are >30 and experienced enough? You get free meals elsewhere also.

I've turned down Google recruiters multiple times. The one time I started going down the interview process at Google for a position in London, the experience was so bad that the recruiter got the technical interview set aside (the guy interviewing me would have reported to me had I taken the position, and frankly based on our interaction, had I been interviewing him I wouldn't have hired him - that's not a good first impression), and invited me to do a second round interview.

I turned her down because I'd gotten a really bad impression of the whole process. No other company I've interviewed at have managed to be nearly as Kafkaesque in the hiring process, and several of the Google recruiters I've spoken to over the years have vented their frustrations about the process at me when I told them this, while non-Google recruiters have gleefully told me they hear this a lot and consequently see less and less competition from Google for candidates.

I'd consider a request from a Google recruiter for an interview again, but the threshold for me to bother starting down that route again has gotten higher each time - I don't feel Google is worth the hassle unless they were to approach me with something exceptional.

Another Google SRE chiming in here... there's a reason why I swear half of the flat vertical surfaces in our office are whiteboards. Including some of the walls.

I'd estimate that no more than 50% of job is spent coding, if that. Solving computer/software/system/architecture problems, yes. Coding, no.

And you have every right to be grumpy, you're supposed to fix problems engineers create, this is why we don't get mad at you. I feel like there should be no difference between sre/swe. swe's should be responsible for the problems they create. That's how it worked at my previous jobs.

SWEs are responsible for the problems they create - if it's a problem with the code, we'll just send the problem their way with instructions on how to go about fixing it. If a service causes too many problems, we'll hand it back to them.

You seem to think a Google SRE is something like a sysadmin. This is not true.

Well, you laugh but yesterday I googled something and it was down. First time it happened to me.. I tried twitter, facebook, both up. Tried again google and it was down. So well I tried bing and got my answer ;-)

From all the posts here, it doesn't like a sysadmin. It sounds like an underpaid C++ DevOps engineer role.

> people rarely solve problems on whiteboards

I don't know what you do or where you work, but it's no job I've ever had, nor any office I've ever been in. It's not unusual to end up spending most of a day in rooms with whiteboards and a few colleagues working through problems.

It's a covert form of ageism. The more questions you ask that are like undergraduate puzzles and less like real world situations, the more likely it is that your "objective" recruitment process will filter for recent graduates.

Is this true? It seems, from the comments from the SREs in this thread, that these puzzles are highly relevant to this functional role.

Saying it's agism is ridiculous. Google SDEs here, the amount of time I spend on whiteboards solving design problems with my teammates very often out weights my coding time. Whiteboards are incredibly powerful tool.

You don't hand a car engineer manufacturing tools on day one of making a car, you need him to produce a design/blueprint first.

Companies rarely ever hand automotive engineers, or other types of engineers for that matter, manufacturing tools. Software is fairly unique in that the engineers are actually creating or modifying the actual production products.

When you work on a large scale problem you WANT to get a consensus of the large picture first, and that's why whiteboard is so handy. We don't jump straight into implementation since it's very often trivial and it's always much easier to improve and refactor implementation than the design/interface of a system itself.

Yes, I agree with you. My point was that software engineering is unique in that they are the ones who end up building the implementation. Automotive engineers never touch the implementation. Their only product is the design, schematics, and blueprints.

The amount of time you spend writing code on a whiteboard outweighs 'coding time'?

Really?

Let's be clear: whiteboards are a fine tool for design, problem analysis, architecture, etc. etc. People take issue with "write me bubble sort up here on this whiteboard."

The amount of time we spend "designing" on the whiteboard very often outweights coding, yes.

The implementation part of "software engineer" very often is much easier and sometimes even trivial when compare to the design/architecture of the entire system. Implementation is also very easy to improve upon and refactor out, if you have a good design to begin with.

I don't think there is any real doubt that Google's application process favors bright young people right out of (usually very good) schools, rather than people with 20+ years of experience. And, that is fine because they are a business and have found that this process works well for them.