Someone finally said the quiet part out loud about ‘Devops Engineer’ as a job title. Only a matter of time before we wise up about SRE as well, I suppose.
I don’t know why we need to have a job title treadmill for this; I hate not knowing what your definition of “devops” or “SRE” is when interviewing. (Both as a person who interviews others and is interviewed by others).
Before anyone says it: Sysadmins could code (not to the same level as feature folk), shitty operators pretending to be sysadmins couldn’t.
Or the company wants every team to add time-series DB-backed monitoring to everything now, and you have to use the same tech stack (no matter how good/bad) every other team is using, and you have to add a bunch of stuff even if there's no actionable thing you can do if a number crosses some threshold or what have you, because again, you don't actually own any of your deployment infrastructure. At best it can complement your regular application logs for debugging some issues and noticing trends (good or bad).
It doesn't have to be all bad though. When specialization works well, it works well, and there's at least a minimum level of service you can expect (even if it's not the best) without having to work for it yourself like you would if you owned all that extra stuff.
There is a legal requirement (regulatory, but carrying force of law) for some industries to implement ITSM practices (and similar, don't quote me on specifics) . There is a requirement in those practices that Developers not have access to production, and that Operations have access to the code. That's incredibly wrong. It's misguided in the worst possible way - The point is to make sure the two audit each other, but it requires black box auditing, when you actually want whitebox auditing. (Note that allowbox and denybox are not acceptable substitutes here).
SRE is called SRE because of a difference in those practices. DevOps is an inexpert redevelopment of those practices. Sysadmin practices evolved into both, but what's modernly called Sysadmin is descended from the AD and Exchange people, and have bad practices. You can't walk back the evolution of words, you can fix them through evolution as well, but it's as slow or slower than getting there, because the ecological niche is already "filled"
DevOps itself as a concept was born in nebulous circumstances (“dev-ops days” being where the verbiage comes from but the founder of that conference called the job “agile systems administration; and the concepts espoused by the devops movement being almost exclusively borne out of the “10+ deploys a day” talk from Flickr).
Anyway, SRE is not materially different than Sysadmins except in three dimensions:
1. Hire only programmers, none of those operators who click buttons.
2. Treat reliability as if it is its own feature.
3. Solidify the contract between feature folks and people focusing on reliability.
I’d like Ben Treynor-Sloss to weigh in here as he likely knows best, but that’s the most condense version of what I understood
You’re right about the exchange people, but they too suffered title inflation, the exchange folks used to be called IT technicians.
The people automating AD deployments across sites and managing reliability were sysadmins, and they programmed in the most ugliest of languages to achieve that, autounattended.xml and bat files for days.
The tools are better now, but the work that devops/SRE’s do in most companies today is why sysadmins used to do in 2008-
Yeah, that's a really good point.
> SRE will not be the title for much longer, and DevOps was a bad title from the beginning and a flash in the pan but it was the title from 2010-2015.
I hope the terms become better defined, not less, though. What do you think the next title will be.
IMO "software reliability" shouldn't and can't be thought of as its own feature. Reliability is part of parcel of every feature. You can't make bad software reliable and good software doesn't need the "reliability" tacked on. This mindset (similar to thinking of "quality" as an add-on from the "QA team") seems very problematic to me. At the very least it's an unnatural way of getting there, at worst it just doesn't work. Where I would draw the line though is the reliability of the infrastructure, that's not the domain of the software developer, so it's totally OK to delegate that portion and draw a contract (e.g. in your item #3).
It's also "Treat reliability like a software engineering problem not a process/operations problem".
Though the majority of efforts were around making the initial designs robust and with as few moving parts as possible; sometimes automation efforts caused more outages than the dead simple operational problems.
(see also: split-brain with pacemaker/corosync on replicated databases)
Hmm...opaquebox and transparentbox? clearbox?
DevOps started as an idea that the development team should be responsible for operations. Before this, most dev teams created artifacts that got handed to an ops team to deploy and be on call for. That idea went to corporations that wanted to modernize, but you can't just disappear an entire workforce of admins used to doing things differently. It's a similar situation to where graphic designers started being UX designers. These people didn't magically develop a different set of skills...just a different set of expectations.
SRE-SE and SRE-SWE's are responsible for application code and often embed on application teams to bolster either code or system performance or both.
Please do not take companies bastardizing these practices as truth to what they are. There are companies who do this right and we should champion them above the garbage.
Put differently, I think one of the ways somebody goes from the "shiny/trendy and unstable" side to the "boring and stable stuff" side is by experiencing the operational pain of their choices. If the pain falls on others, will they still learn?
Of course, the way you talk about your job makes me wonder if you are already experiencing so many systemic/managerial issues that there the feedback loops are already pretty broken, so this one may not make a ton of practical difference.
Oncall is a scourge not because of the experience of technical problems, but because people already working full time have to arrange their lives outside of work around a second "oncall job". A job which occurs after hours, one out of every X weeks.
A dedicated, pure "Ops" night shift (perhaps in another time zone) would be more humane.
In my experience, this leads to design that pushes problems to outside of working hours.
"We don't need to fix that edge case, just have the off-hours ops team do a manual workaround every now and then."
Or "What does it matter that the deployment is error-prone? We can just schedule it with the off-hours ops team."
I think that depends on the seniority of the individual/team. In my experience, of course one can still learn.
To give you a real example: years ago one of our systems went down on a Sunday morning and our team had no oncall people. The infrastructure team was the one who fixed the issue (don't remember the exact underlaying issue, but it did make clear one aspect of our service we didn't properly: signal handling). Next morning the team wrote down a Jira issue to improve the way we handle signals. Ticket got prioritized very high and was fixed the very same day.
Now, what would have happened if the issue that Sunday morning was due to a bug in the software our team wrote? The same thing. The difference is that infra team would have no clue on how to fix the thing and would have to revert the service to a previous stable version. Would the business be fine with it? In our case, yeah. As a matter of fact, they didn't want to spend the extra money hiring ops people for each team to be on call. You see, if the business really cared, they would immediately have hired a software engineer willing to be on call... They just didn't care that much (and they couldn't force the current team to be oncall because our contracts didn't specify so and the average age in our team was around 35, and nobody wanted to be on call).
But how did the senior engineer learn to handle those situations in the first place?
Personally, if I get paged at 3am due to a bug, I'm going to fix it regardless of what the 'backlog' and 'prioritisation' and 'sprint goals' and 'feature roadmap' and 'product owner' say I should be doing.
But some would say I should not be bypassing the process in that way, and that the feedback loop of external stakeholders making requests to the product owner is more than sufficient.
My company goes with option 3 from the list, “It’s not part of the job outside business hours, but we might still try to reach you during those times.” and it's working fantastically for us.
One out of eight weeks my only job is handle alerts, incidents, and questions from other teams. I and my coworkers dedicate this time to burn down technical debt and to add documentation to the codebase. This keeps error rates quite low on its own but we combine this with a scheduled release cadence (3 times a week) and a reasonably sized QA team that tests the major feature flows of our product for each release (~1 QA per/ 25 engineers) and an ops team whose job is built around being available to rollback to a previous release.
Even though we're a large company with millions of DAU most teams get pinged out of working hours less than once a year. My experience here has really pushed me to the opinion that continuous delivery has been hugely destructive for most of the industry, eroding our pool of experienced engineers who can't be bothered to do on-call.
Work has had three out of hours pages in the last two years, all self resolved within a few minutes.
I’m more focused nowadays on “what problems are you hiring me to solve?” since it feels more and more like the Venn diagram of the three job titles has nearly completely coalesced into a perfect circle.
Difference for me is I’m scrutinizing far more intentionally in job interviews about why an org is hiring for SRE/Devops before accepting any offers. Too often orgs are hiring for this talent and turning them into kitchen sinks for anything and everything the SWEs aren’t doing.
Compliance? Send to Devops.
Upcoming audit and need a pen test done in 3 days? Send to Devops.
Did a bad job prioritizing bug fixes and now shits crashing? Devops.
Etc. once you go through that a few times you start to figure out the right questions to ask in an interview and figure out if you’re about to join a company with Devops practitioners or pretenders.
- Why are you hiring Devops/SRE?
- What is a Devops/SRE going to bring that isn't/can't being done by engineers presently?
- Why isn't it being done presently? What have you tried so far?
- How many other SREs/Devops do you have? When will I get to interview with them (if applicable)
- Who is responsible for platform? Infrastructure? Deployments? How are they involved? When are they involved?
etc. As mentioned in my last comment, a lot of it comes through the baptism of working at a lot of really crummy shops to know the kind of bullshit you don't want to put up with. You gotta deal with some of it no matter where you go, but you sure ain't gotta deal with it all.
This is a lot of boilerplate stuff, sometimes you're lucky and these questions get answered before you can ask them, sometimes they're in the job description. So let me talk about that for a minute.
You really want to take your interviewing to the next step? Learn how to inquisitively, but tactfully challenge what you're reading in job descriptions. The answers I've gotten have been far more revealing than "what will I be doing day to day?" if you ask for more details about a bullet point or two and why those bullet points matter, or who they matter to. That includes, yep, on-call.
Most of my other questions are very probing questions about things in the job description; not necessarily because I'm looking for a specific answer, I want to see how the hiring managers and others describe those topics. Can they actually talk about why they're looking for someone to do x, y and z? Can they have a meaningful dialogue about what those responsibilities mean for the team or are they just parroting back what the job description says, like someone in a zoom call just reading words off a powerpoint slide?
Here's an example:
Job says they want a Devops to come in and also be responsible for security, risk and compliance in the infrastructure? Okay, here's my counter-inquiry about that: if Devops has the responsibility for security, risk and compliance, talk to me about the authority Devops has to recommend or deny certain actions in the platform if it is assessed to be too risky or costly to maintain a compliant and secure posture were we to do it anyway (if you've ever been in that unenviable position, you probably know exactly what I'm getting at with this question).
Interviews are two way streets, and in my thirties with a family where "family time" has no fungible cost, I'm driving very defensively on my side of the street.
Interviewed with dozen of companies over my career - never been able to get a straight or truthful answer to this