Hacker News Reader

Comment by no_wizard | Hacker News Reader

I find this bit to be particularly insightful:

"Can I handle this? What if I can’t?" But then I started to work the problem in front of me, like I was trained to, and I remembered that I don’t need to know everything — there are other people I can call on, and they will answer. I may be on point, but I’m not alone

It might be because I'm currently training people in this realm, and this is one of their biggest fears, or maybe because it was my biggest fear, but its so true. We're a team. We're here to help. At least if your SRE org is any good. Never be afraid to ask for help, and never be afraid to admit you don't know something or it might be outside of your comfort zone.

I'll take willing to learn and readily able to admit knowledge deficits over someone who doesn't any day of the week. Great book they're working on, great article on this. So many gems, but this one stuck out for me, and its pretty relevant to me right now.

View on HN

I notice this alot with the team I work with. Our team's tickets are not assigned automatically: there's simply a queue and people are encouraged to grab what they can. Unfortunately, what I've often seen is that people see something in the description that they're not familiar with and refuse to touch the ticket because they don't want to ask for help, which means that it languishes in the queues. The end result is that the same person ends up taking the same kind of ticket over and over because they're the only one who has any familiarity with the program in question.

I've had similar experiences in the past. In my situation it was because the Seniors were.... un-helpful to the Juniors, to put it nicely. This is where having a solid group of helpful Seniors can help everyone work better.

I've seen this be the case in other places. We used to have a queue based system. Then we went to an auto-assignment process. One of things it has done is open up communication on our team, since when we did it, we implemented as a rule of policy that if you aren't familiar with something getting assigned to you, you first ask for help before trading (we have a formal mechanism for soliciting this).

1. It encourages everyone to be aware that we're here to help

2. It encourages learning, the number of request for help decrease over time once the experience and familiarity ramp up

3. It exposes everyone to different types of work.

There are exceptions of course. Our P0 bucket has a dedicated set of people that handle that, and they are hand picked because those are house on fire situations that need the experience. Its also the one we put the juniors on when they know the ropes and are ready to take on the critical tasks so they can advance themselves (its good experience for career and personal growth I feel).

The other thing i like about this is that as a manager, I can actively encourage behavior by claiming a ticket, or helping someone with a ticket. I really want that culture effect to happen from the top down.

It's not mentioned in the article, but there is an underlying point that affects hiring for roles like this: you need people who can and will admit they don't know everything and will ask for help rather than wing it.

"Rock stars" are downright dangerous, as are people who prefer to make things up rather than admit ignorance.

A new SRE doesn't need to know everything (and can't). But he absolutely needs to be curious and willing to ask for help.

It's an interesting dilemma in my mind. I remember reading through the SRE book about how SREs a required to have both depth and breadth. Seems like a nearly impossible target to hit IMO, so how are you supposed to simultaneously reconcile deep/broad abilities with humility when hiring for SREs?

You don't, you just wait and struggle to find the 0.00x % of people who fits the role.

There is a reason that this is an impossible to fill role.

Our litmus test is this, I don't know if it gets to the heart of the question but I feel it does.

We look at essentially, (but not unequivocally) these things

1. Proven experience and desire to learn is a must. One of the best i have ever worked with came from a place where they worked pilot projects, and had to manage all their infrastructure themselves as the developers. No certs, no formal SRE experience (IE, thats not why he was employed previously). One of the best. He loved the work, and I could tell he learned so much doing this. It doesn't have to be this extreme, but having a proven interest in this line of work is top priority.

2. Is their depth better than their breadth? It is correct that you need a LOT of breadth, however I value the depth first. I'd rather someone have say, a medium about of breadth on the different technologies out there and a lot of depth on core subjects, like container management (this happens...everywhere nowadays) or cluster management. I don't need you know every single implementation of this in depth though. I need you to at least know one implementation of this in depth. I can build on that.

3. Because of the first 2, I need someone who is team oriented, as always.

(I'm an SRE at Google. My opinions are my own.)

> core subjects, like container management (this happens...everywhere nowadays) or cluster management

Curiously, these are subjects which most Google SREs won't know much about. One team deals with all that stuff as a service so the rest of us can get on with something else.

What would I pick out as our core skill sets? Ignoring technology-specific details that won't apply anywhere else: troubleshooting a system that you don't understand (reverse-engineering it as you go), and non-abstract large system design.

I always reconciled this as a T shaped graph instead of a box. It's important to have depth in the service area you can control and a breadth of understanding across your dependencies since Google builds service oriented architectures.

Mostly it meant SREs tended to be older than other engineers at Google (I was an SRE there for a while), I think by an average of nearly a decade.

Broad experience, depth on a few topics. It's not impossible at all, it just takes time.

(edit: Note this may have changed since I left the company in 2009. It's been a while!)

Depth + breath is another way of saying experience. Program long enough and you'll learn all sorts of little nasty things about garbage collection and permissions and tcp packet headers and faulty JSON parsers. All of it crystalizes into those little moments when you think "I've seen this shit before".

The hard part is to know what you know and what you dont know. It has nothing to do with "inside a team and feeling safe". If you do not know what you know and what you dont know, then you will make the most naive mistakes eventually.

I myself have deleted the data file of a production mysql server, because I have no idea what I am doing. I need to call teammates in 12am to learn how to take care of the mess I created.

I agree, it can be complicated. Thats why i like the auto-assign ticket system (and not putting absolutely new people on P0 duty until they have warm feet). It allows my team, at least, to experience breadth of issues and if one because of time, or because of efficiency it can be traded in relative real time.

However, having it be policy to ask first then trade or have someone else own it means essentially as a team, we can own that issue with you as you learn. It helps a lot come crunch time.

Mistakes will always happen though. If they are genuine and not from a lack of caring or trying with an earnest/logical thought process behind it, it doesn't matter in the grand scheme. I'd rather have my entire production server go down on an error like that than deal with a situation where the person was just being negligent to their duties and wasn't willing to be humble.

In short, you'll be fine :)

I agree about the willing to admit you don't know something. I've found it's something a lot of people have a hard time doing.

I wrote down some thoughts about this recently: http://zalberico.com/essay/2017/02/21/asking-questions.html

Goes for any job really; know what you do/don't know, and know what you should/shouldn't know.

For some things, you should be able to trust others.

It's funny, because it boils down to "don't lie" (to yourself nor to others).

And this, is why the current US president scares the shite out of me. There are a bunch of jobs with a breadth and depth requirement where this is the reality. The first people liars lie to are themselves. (and I'm not being glib- I've done a heap of fraud cases) How does that person ask for help? Not knowing their limits of competence.

>there are other people I can call on, and they will answer. I may be on point, but I’m not alone

that sounds really nice

Reply on news.ycombinator.com