Ask HN: What makes SRE great compared to "plain" DevOps? | Hacker News Reader

Ask HN: What makes SRE great compared to "plain" DevOps?

10 pointssamuell17 comments

In my view it is having a dedicated team focusing their full mental bandwidth on pro-actively understanding and managing robustness of the system.

In Pure DevOps, it seems to me developers often don't have the full picture of the system, and not enough bandwidth to foresee complex interactions from their changes.

These are from my experiences spending one year as a developer in somewhat large a greenfield development project for a machine learning infrastructure based on Kubernetes, where subtle platform errors would often cascade into the applications layers, causing really hard to debug problems that often left more junior application developers scramble to even know where to start debugging.

I'm preparing a presentation though, and I'm eager to hear your experiences and perspectives on this!

(SRE = Site Reliability Engineering - https://sre.google/ )

View on HN

Comprehensive knowledge of the application codebase(es) you’re responsible for and the ability to quickly determine root cause of failures and revert or remediate, either by shipping hot fixes, infrastructure changes, or both.

Yea, this is at the heart of what I also thought were the very point of it.

In my experience on the mentioned project, I felt one of the few ways to get there, unless you wrote the whole codebase, is to have enough time to build (or configure existing) some tools to be able to visually understand and/or navigate the system.

During one week of being on-call engineer, inspired by the ideas of pro-activity and monitoring in SRE, I stole myself some hours to put together a script that would parse our ArgoCD configs to create a map of our Kafka topics, into a GraphViz-based HTML file with links to to the relevant topics as well as microservices (pointing to some web based inspection tools for these).

This finally made me be able to figure out what was going on and be useful during incidents as a newcomer in the field, but no one in the team had thought of this idea of being a little pro-active, so most of the time when errors occurred, people would just scramble around all over the place, and only the really senior guys would be useful because of having the whole system in their head anyway.

To me, this idea of pro-activity is what SRE brings to the table.

Doesn't that mean it's essentially a senior developer in the dev team?

It does, but a senior developer in a pager rotation.

It makes sense. But I do feel seniors will be pigeon holed into taking an Ops responsibility.

Depends on the org. In an org where devs are protected, DevOps/infra team will do the pager duty suffering in an SRE capacity. In other orgs, seniors will be forced into it unless they have leverage to bounce otherwise, or they tolerate it because there are worse jobs out there. Culture and appetite on both sides is highly variable, won’t know until you’re on the inside.

It definitely can be, but depending on the application, it might make sense to demarcate 'scale/infra' related senior developer responsibilities from 'business logic' type responsibilities. An SRE would care about the former, much less the latter.

I'm a SWE SRE, and I don't particularly care to look at the code of individual features being added unless there's potential for a large blast radius, but I do care to make sure if something breaks, there isn't cascading failures. I want things to degrade in a predictable way, or to fail in an isolated fashion. The non SRE SWEs are free to break a single feature, and i'm happy to help them roll it back, but that's not what I'm primarily worried about.

Conversely, many of them aren't all that interested in thinking about how slowdowns or failures propagate upstream or downstream to other teams that may or may not understand how things are connected, whereas I do worry about that.

SRE brings a laser focus to system reliability, ensuring proactive management and a comprehensive understanding of the system's robustness.

In comparison, 'plain' DevOps might sometimes lack that specialized attention, potentially leaving blind spots in foreseeing complex system interactions. SRE is all about that extra layer of assurance for smoother, more resilient operations!

SRE and DevOps teams overlap a lot, with SRE being a specific implementation of DevOps, more focused on reliability and scalability by applying software engineering principles.

Sometimes traditional sysadmins will be labeled as DevOps because they are using some config management tool, but the practices are the same.

The next level is Platform Engineering, which creates a platform that encapsulates all these best practices.

You can have dedicated or distributed DevOps and SRE teams. You can't have distributed Platform team.

My latest job title was platform engineer. Honestly it felt like a hack of all trades role. I was doing everything from k8s tf etc deployments observability monitoring to writing backend services running analytical queries and building UIs while managing their entire lifecycle. Really fun, enjoyed the heck out of it because every day I was learning and doing something new and interesting. 11/10 would recommend. Not sure if that’s what platform engineering is generally considered to be.

I think that's it but what matters is if your team was exposing a platform that made the hugely complex infrastructure more amenable to developers, while helping people to stay on the golden paths of reliability, monitoring, etc. It also helps the organization to enforce some standards centrally.

What happens behind that interface can be chaotic for sure.

They'll all be washing dishes when the next recession hits IT and the need for affordability exceeds the need for reliability in service.

When the first company relearns that on-prem with dedicated admins is infinitely cheaper than cloud with similar reliability and less / cheaper staff overhead it will be a rude awakening. This is especially sad to me as most women in our field are in DevOps/SRE roles as opposed to SWE.

Pray that Bezos and co. can keep their customers / your employers fooled.

Though I understood one idea with SRE is that they are much more of a developer role anyway, and should theoretically have an easier time switching to a dev role if need be?

How is the role related to where it is deployed? Won't this role carry over to on prem deployments?

Biggest load of horsecrap I have seen on this website. Complexity ballooned into unimaginable proportions. Observability is crucial for complex software.

That's my reaction too. I cannot imagine, unless we experience some world-wide cataclysm, moving away from observability, ci/cd, configuration management etc. and going back to bespoke bashscripts, hosts-as-pets, builds needing whole teams to put together, and whole teams to push out to servers.

I experienced a reversal like that once, going from a company with a very healthy SRE culture + technical setup, to a startup very much without it, and I couldn't believe we were wasting our time on manually doing things and putting out the same fires.

(I and a bunch of other SREs were hired by the CTO to fix this. He left 3 months later and my group was folded into OPs. Each one of us quit within a few months of that)

What makes a great apple compared to "plain" pile of poo?

One is a role, the other is a broken mindset.

Reply on news.ycombinator.com