Hacker News Reader

Comment by peterwwillis | Hacker News Reader

I don't know anything about Google and its SREs. Do they really just hire software devs and expect them to be good at Ops? The opposite, hiring a sysadmin and then assigning them to a software team to develop a product, seems equally problematic.

View on HN

They want a cross between a developer and a sysadmin.

When I was there they hired people with one background and hoped they would be capable of the other. This didn't work out for about half of them. Making things worse, they had no procedure to say, "OK, we hired someone we think is good, but this is clearly the wrong role for them." Which lead to losing a lot of people who had potential.

Making it personal, I interviewed for a pure dev position then was offered a job as an SRE. It turns out that I don't make a good sysadmin. After I left, I was amazed at how many people I met knew of someone else who had had the same experience.

Ironically recruiters see a year as a Google SRE on my resume and think I might want to join SRE teams at other companies. Most of that stopped after I changed my Linked In profile to say that I am never interested in SRE jobs.

Yup. Exactly my experience.

Just curious, what made you think you weren't a good admin?

Explaining the concretes would be a lot of unpleasant detail. The why is far more interesting.

I am very good at "tunnel vision". Taking a thread and following it in depth. Completely learning a system in depth so that anything that comes up, I know exactly where to go and what to do.

I am weak on "peripheral vision". Keeping track of 15 balls in the air, and operating with limited information about each. Being effective with limited context about each individual system because there isn't time to learn any of them in depth.

Tunnel vision is a virtue in a software developer. Peripheral vision is essential for a sysadmin.

Now imagine a person with poor peripheral vision trying to learn a system as big and complex as Google's. And being responsible to support 15 different pieces of software written by 15 different teams so that when anything went belly up with any of them, you can trouble-shoot and get it running again.

Nothing particularly bad happened, but I also wasn't accomplishing the job to the standards that Google wanted in that role. And I was not the only person on my team failing in that way.

Ideally it would have been my manager's job to say, "I recognize that this person isn't working out here, is there a better role for them?" That didn't happen. Several months later my manager got fired. I was privately told that my situation was a trigger, but I don't know details.

The fact that my situation is fairly common strongly suggests that the ultimate failure was organizational and systemic. I don't fault Google for hiring developers into their hybrid developer/sysadmin role. I do fault them for not having an explicit onramp/reconsideration process to mitigate the risk that they create by doing so.

Well that's just the thing - an organization that has a "system" or "process" for handling employees and their work is by design less than flexible. In a small to medium sized business, you might have 3 completely separate roles, and others might pitch in as needed. But at a large corporation, it's simpler and more efficient for them to have one person who does one job. When the peg no longer matches the hole, they are replaced or reassigned. Which is sad, because people with valuable skills are often underutilized. I feel like companies like Valve might have the right idea going forward.

The problem is that larger organizations actually need to have a defined process of some sort. There is a point beyond which individual judgment doesn't scale.

In general I believe that Google has a pretty good process. However every process has bugs. And I happen to have encountered one where they pick people who are qualified in one role into a more experimental one, and it only sometimes works out for them.

It's notoriously hard to hire for. They cherrypick devs that are 85-99%[0] of the way to passing the Google SWE hiring bar, who in addition have ops experience.

The Google SRE head honcho says: [1]

>Fundamentally, it’s what happens when you ask a software engineer to design an operations function.

So I think the short answer to your question is yes, they hire software devs and train them [2], but require them to already have some ops/Linux internals knowledge.

[0]: SRE Book

[1]: https://landing.google.com/sre/interview/ben-treynor.html

[2]: https://www.usenix.org/conference/srecon15/program/presentat...

They kinda do. Over time, the skills of a dev and SRE diverge a bit but at the core they are the same. I am a dev at Google but went through a program called Mission Control[1]. (It was a great learning experience and helped me do my job as a dev much better.)

As others have said, there is a pretty intensive training that you go through and obviously you learn a lot when you are handling things...

[1] http://googleresearch.blogspot.com/2012/07/site-reliability-...

You're quite right. I'm an ex-Googler and SRE was the last thing I did at Google. As a first-and-foremost programmer I was bored out of my wits and frustrated by the fact that while I was writing repetitive monitoring rules actual development was happening somewhere else and I had little impact on the design of the system. Granted, a lot of folks over in SRE loved it but I would venture to say that most of them were much closer to being a sysadmin than programmer. Folks like myself tended to 1) move to SWE, 2) quit and 3) some endured and became managers over time.

Although I can't find the link now, I've read that there is a pretty intensive training process for SREs. This would make sense since Google's infrastructure is so unique.

I think you are right. There is a lot of training after one join the team. The job does require very different skill sets.

From my limited experience, they want devs. The initial phone screen is all sysadminy questions (e.g. what commands show you i/o usage?). The second screen is a coding interview. I'm a sysadmin who works in bash every day. The coding questions are mostly related to log parsing. My code interviewer didn't seem to even acknowledge that bash was a programming language and did not understand a simple grep and sort pipeline.

Edit: Sorry everyone! I totally got my LinkedIn and Google interviews mixed up. What I described above is my LinkedIn experience.

There are actually two different SRE roles: the one people are describing above where you are 85-99% of the way to SWE (Software Engineer), and you have sysadminy experience, and another one where you are 100+% of the SWE bar and optionally have sysadminy experience. The former is called SRE-SE (Systems Engineer), and the latter SRE-SWE.

SRE-SE interviews are super heavy on the sysadmin stuff usually, with less (but still significant) attention paid to SWE skills, whereas SRE-SWE interviews may not even have an SRE component (it's possible for candidates in the 'normal' SWE hiring pipeline to be shunted to SRE-SWE post-interview).

Yeah a lot of people don't understand this distinction. You have your pure SWEs who were hired that way who then were either picked for or switched to SRE-SWE. Then you have people who were recruited into SRE-SWE from the beginning. People in SWE and SRE-SWE job classes can freely move between them. Then finally you have people who were recruited as SRE-SysEng, or were recruited as SWEs and didn't quite make the cut. These folks have to do a transfer interview to jump to the SWE or SRE-SWE roles.

Could you speak to the dev/programming skill set need for the SRE-SWE vs the SRE-SE?

I'm an SRE-SE and regular do phone interviews for SRE-SE candidates.

While I do tend to spend more of the interview time talking about sysadmin tools, operating systems, networking, databases, security and troubleshooting, I still expect candidates to have reasonably good coding chops.

The difference is that the coding questions tend to be more task-oriented or procedural (i.e. log processing, building automation pipelines, implementing standard unix cli tools, etc.), rather than the algorithmically challenging or math-oriented problems that we'd usually ask SWE candidates.

Both the SE and SWE side SRE candidates need to be able to design and reason about large systems, making trade-offs between performance (especially latency), redundancy and cost.

Thanks. Is being well-versed in C a prerequisite for the role then? I'm imagining you need to be fluent in at least one statically compiled language or ???

In my interviews you code in whichever language you prefer. Some interviewers will ask you to use a specific language that's mentioned on your resume. In general I think that if you show strong coding skills in some language, it is believed that you won't have much trouble teaching yourself the languages your team uses (typically some subset of C++, Java, Python, Go, Borgmon).

Weirdly my experience with interviewing for an SRE role was the reverse: the phone screen was pretty much a straight programming problem (in a Google Doc, ugh) but then one of the on site interviews was very shell programming focused.

I guess to a certain extent it must be the luck of the draw - mass log analysis did crop up in my on-site interview as well, so that seems to be something of a theme, although it was more from a high level systems building POV for me.

I interviewed as SRE four months ago and I got a very typical programmer phone screen and interview, just exactly the kinds of whiteboard CS questions you expect Google might ask. (My bigger strength is on the programming side instead of the ops side, so I think perhaps they explicitly tailored the interview to measure that.)

That's interesting, when I interviewed (admittedly it was like 8 years ago), the first interview was standard CS questions and the second interview focused on deeper C++ questions (things like analyzing how a vtable is constructed).

I don't work for Google, but I did follow that career path - started as a developer, then moved into operations. I didn't immediately become an expert as soon as I made the transition - it was a slow process, and I'm still not what I'd consider the equal of a veteran sysadmin in terms of low-level system knowledge. However, I can hold my own in either discipline.

They look for someone that is strong at operating systems and networking knowledge AND someone that's a decently strong programmer. IOW, someone that can pass a low-level ops interview and a Google SWE interview.

This is a tough position to fill but I've worked with folks who are genuinely good at both, so they're out there.

What I'm wondering is have they created a model that only works if you employee "top 5 percent" people.

Sort of like if long-haul truckers needs the skills and training of Fighter pilots.

Yes. What you get is a lot of "grunge work" that never gets done because everyone is busy inventing the next gen cure-all.

> Candidates should be able to pass or nearly pass normal dev hiring bar, and may have some additional skills that are rare among devs (e.g., L1 - L3 networking or UNIX system internals).

They hire devs with ops experience.

Can you or someone else possibly say what the "normal dev hiring bar" is?

I've enjoyed what I've read so far in the book and I get it that there is an opportunity here for Google to market the SRE role b/c they are apparently hard to fill but why didn't they put a chapter in the book of what that dev skill set is?

Or else what dev skill are expected of someone working in this capacity.

Seems like lifting the shroud of mystery would do wonders for marketing a role that is hard to fill.

SRE-SWEs at Google are expected to be able to design and implement the same things that the developers they support are working on. If you are on Blah-SRE you would be expected to be capable of at the very least reading and understanding the code that Blah-Team writes, reviewing their code, etc. You might not _actually_ work on that code (it might be more important for you to be working on the automation, monitoring, or whatever) but you'd be qualified to do so. That's why the SRE-SWE interview is basically the SWE interview: graph coloring and complexity analysis and all that.

In C, Java, Python?

Good question :) SRE-written system are never or almost never written in Java. However many SRE teams support Java systems.

Literally nothing at Google except third-party code (like Linux kernel) is written in straight C. C++ is very common, though.

Python sucks and is useless. Most things that were written in python are now written with Go.

I guess your question is about what happens if a C++ expert ends up on a Java team, but I don't really know. I happen to have landed in C++ world, which suited me.

> Python sucks and is useless. Most things that were written in python are now written with Go.

This is simply not true in any way.

True in my area as well. The only new things being written in Python are things that need to interop with other systems that have giant existing Python libraries.

Well it's a big company. In my area people who propose writing new code in python are either gently corrected or heartily derided.

I was an SRE who was most comfortable in C++ who changed ladders to a SWE team who was primarily Java. To be honest it was harder to learn all the team specific stuff than pick up Java.

The reason Python tends to suck is not all really nice libraries are ported to it in any reasonable amount of time. (If you need it, you end up doing the work). Go sucks in new and totally different ways.

I'm imagine an SRE needs to be fluent in at least one statically types compiled language. Would Go fulfill that in terms of getting an interviewed or hired or does it need to be C/C++?

Shouldn't matter. You should be capable of picking up new languages. You do the interviews in whatever language you prefer.

Can't be good at everything. Only so many hours in the day.

Reply on news.ycombinator.com