Hacker News Reader

Comment by pram | Hacker News Reader

I worked at a major software company that was implementing an “SRE” org and it was frankly pathetic. It basically had its genesis in some higher up hearing that Google does this thing, so we probably should too. That then later turned into a bunch of random DevOps/Sys Admins drafted into the new org reading the Google SRE book. It was a shamelessly cargo cult approach.

After a year all we really had was hundreds of Confluence pages specifying policy for this and that, a bunch of ceremony added around releases, and arbitrary SLA/SLOs defined. Almost zero actual technical work was accomplished. They then started filling the team with new grads who had neither devops or sysadmin experience, kind of cementing the perception that it was a non-technical role.

Either way I’m not a fan of the entire concept after being subjected to that suffering!

View on HN

Blindly replicating what Google is doing is really problematic, I saw something similar at a few companies. The principles are interesting, but they have to consider each company's reality.

Google is putting a lot of marketing money on the topic, but from what I saw they are trying to alert that there is no one size fits all for this.

So far I haven't seen any companies replicating what Google SRE does, blindly or intentionally. Much as this article describes, organizations I've seen are just inventing a role they perceive to exist in their organization and slapping the label "SRE" upon it. Often these roles are directly contradictory to the way Google SRE works: they are just carrying pagers for products they aren't allowed to improve, they own build/test/release for some reason even though that clearly falls in the domain of the developer, SREs are setting service levels which is obviously the responsibility of product.

I think people should realize that when you go into the hiring market looking for SREs, you are about to pay more, not less, for a highly qualified software engineer who is also a systems expert. You want these people to not just operate product, but to be closely involved in architecture and development so the product turns out to be operable. Your SREs should be working in the product code on the bits that ordinary developers might overlook or not be qualified to write, because they lack expertise in system or distributed systems behavior.

What this article is calling "real world" is just an attempt to redefine the term.

An SRE with no authority to do anything is just an overpaid totally frustrated paperweight. They have to be able to change things on both sides of the 'fence' if they can only do one or the other or neither then their role is not an SRE no matter what sorts of titles you give them.

Any org I go into that says 'we want an SRE because google has one' I can not help myself I blurt out 'you are not google. what works for them will not work for you, you need to find your own way of doing this that fits your org. Trying to mimic google without the understanding of why they way they are will damage your org'

>>> An SRE with no authority to do anything is just an overpaid totally frustrated paperweight.

That matches my experience at most companies, the SRE/DevOps is just the sucker who is on call.

I think authority is a critical differentiator between the SRE described in the Google book and SRE as it’s practiced elsewhere. Google SREs can impose moratoriums on feature development or deployments if they judge that a product is becoming critically unreliable (which has a well-defined meaning based on SLO breaches). SRE teams will also sometimes “offboard” a product that is insufficiently respectful of the finite attention that an SRE team needs to spread across multiple products, returning full production control to the development team along with the pager and all responsibilities thereof.

I think one more differentiator is the dedicated position. Rotation is not SRE. It can be helpful to help people understand both sides. But just rotating your team into the role every few weeks with no real 'group in charge' just seems to make them mad.

> An SRE with no authority to do anything is just an overpaid totally frustrated paperweight

That's exactly how it is implemented in my company. We are called SREs but everything is driven my management (and by that I mean a single VP who is omnipresent).

I thought an SRE is contacted when your house burns down due to for example scaling and to improve operations where much more load/customers are expected, no? Where the house is expected to fall but where it is not allowed to happen.

No, you bring the SREs in to consult on designing a fire-resistant house before you turn a shovel. Calling them after you've already designed, built, and burned the house is likely to leave you disappointed.

I feel like there is an analogy with self-help books. Every personality and company is different and they have to do the work to find out what works best for them. Internalizing the meaning of things and leading your own thinking is probably as important as the practices themselves.

Adopting another company's practices is often lazy thinking. There are no silver bullets.

Marketing money? besides paying people to publish their work?

and the fact that not all the companies run the same workloads or scale as google

> Almost zero actual technical work was accomplished.

This is 100% my problem with this article.

At no point is it ever implied WHAT or WHY or WHERE actual work is being done. Only HOW they organize it. It looks to me from the article they are using every “cool” tool and seem to care more about the how than the why. And that’s fine if it’s the point of the article which I guess it is, but it seems like it’s “reliably” structuring a husk.

I defend the costs of IT and in this case the costs of an SRE approach - but these aren’t the things that make you money and keep your business afloat.

When I spend a couple weeks making a system wide event bus to get messages from n to m that’s cool and I need it... but customers don’t pay me for that. They pay me to fix a problem they have.

I’m very cautious of articles like this that say all “the right” things but don’t even hint at solving customer problems.

You just stumbled upon same issue google had - sre-swe is really tough to hire for. The concept is good but you need software engineers w/ systems background and they’re relatively rare and most good ones are already at faang

I mean Google could stop being completely elitist with SREs needing to come from SWE backgrounds and hire the thousands of SRE/Sysadmin/Ops types who do know systems very well but have a lot/bit less SWE background and let their SWEs contribute to mentoring them on SWE stuff.

Pipe dream of mine, for sure.

There's been at least 3 G SREs that have posted in here about what they do day to day and it's literally no different from any other non-FAANG SRE/Ops role any of us do at various scales except for the gatekeeping.

Google does hire SRE SEs and I interviewed many - the focus on coding is a lot less and they get grilled more on systems etc. it’s important to have a healthy mixture of both or you run into eng culture issues like described by many posters here

Also, a systems engineer usually has a different way of thinking about a problem then a software engineer and vice versa.

This helps tremendously in troubleshooting and building stable products.

Having a combination of systems, networking and programming "perspective" on a team seems to result in very robust systems all around.

> Google does hire SRE SEs and I interviewed many

For those who aren't familiar with the google acroynm soup: SE here means systems engineer not software engineer(SWE).

Fwiw about half of the sres I know at Google come from sysadmin or help desk roles within Google.

Is this TEAM? There's a nasty issue with companies trying to emulate Google on "all the things" and it usually doesn't work, since often times these organizational structures are a natural evolution of where a company came from, but you aren't that company...

Best SRE/QE model I've experienced was with a couple of these guys being embedded into a dev team (reporting structure didn't really matter), such that they participated in the team meetings and rituals, becoming domain experts. One fellow eventually learned the SAML2.0 spec inside out and was invaluable for catching bugs.

I suppose being embedded but reporting up through a different org did have it's benefit: namely, the consistent dissemination of best practices across the company, but with a judicious eye to applicable context that the current do-it-all-swe cannot with the current SRE ivory tower model.

The do-it-all-swe model has yielded trash QE/observability in my experience.

Yes, lol

Sounds truly horrible, and meets my experience of SOA around 2005, but let's not forget that younger devs don't have the luxury of having tinkered with stuff hands-on as it was introduced, such as SQL, pre-jquery js, SOA, Unix/Linux to form an intuition.

Is that not a problem with the training / education of entry level.

This was the problem the raspberry pi was designed for GCSE and A Level students to actually get hands on experience, so that when you go to to university you had at least the basics.

Reply on news.ycombinator.com