Comment by _vertigo | Hacker News Reader

Comment by _vertigo | Hacker News Reader

Parent Full thread

It does matter - in my experience, the future job prospects of "System Administrator" are much worse than "SRE"/"DevOps".

Switching my title from "System Administrator" to "SRE" within my last company resulted in a job family change and a 10% raise (I had to show I could code as well as a software engineer in order to make the switch).

When I left that role, having "SRE" on my resume instead of "sysadmin" was (probably, I don't have any strong evidence for this) instrumental in getting responses when I applied to "Software Engineer" roles at selective companies.

I think there is a bias against roles that don't code, and especially roles that sound outdated.

A lot of larger companies and managers in those companies don't understand what DevOps is, what SRE means or anything like that. They just know Ops and Dev, and your previous job title is probably the strongest hint they have to work off when they categorize you. Getting lumped in with Ops is (probably) a big hit to your earnings potential and limits your future options if/when you decide to move.

> Switching my title from "System Administrator" to "SRE" within my last company resulted in a job family change and a 10% raise (I had to show I could code as well as a software engineer in order to make the switch).

Perhaps we're in agreement, but this is critically important. When my company hires for DevOps, CloudOps, etc positions, we are innundated with applications from Ops/SysAdmin personas. We don't want people who will put out the fire and keep the system limping along--we want people that can't stand putting out fires _and who have the skill set_ to build systems that (1) aren't likely to catch fire and (2) are easy to troubleshoot/extinguish when they do. One way we're addressing the problem is to change titles from "DevOps" and "CloudOps" to "Cloud Engineer"--not sure yet how big of an impact that will have (if any at all), but it's worth a shot.

In that case, isn’t an excellent DevOps/sre engineer indistinguishable from an excellent software engineer? Why hold any distinction at all?

From my experience, SRE is a person with developer mindset (and skillset, or the desire to have a developer skillset) who doesn't mind touching the infra.

A SWE typically won't want to touch infra. That's my experience of course, YMMV.

SWEs who don't touch infra show up because universities today pump out a lot of book-smart real-world-dumb graduates.

I don't think so. I have my areas of expertise and of course in a pinch I will pitch in and try to help any way I can, but I've always found my best work is done in a team with fairly well defined roles, a healthy respect for each others' specialisms, and an enthusiasm for short bursts of collab/pair coding and longer stretches of solo(ish) work.

I have fond memories of the team that worked across three time zones so I would get up to a set of well-described problems that I would solve in the morning (so satisfying), then a nice stretch of feature building after lunch, then a burst of pair coding with my newly arisen colleague, then maybe finishing with writing up any roadblocks or requests for the next person. I got very used to identifying blockers that were out of my area and would be better solved by the domain expert, and also a LOT better at ticket writing. It was a really, really productive and rewarding workflow and one of the key points was not getting bogged down in stuff outside my areas. We also all really appreciated each other because we all experienced each other as magical elves giving answers to hard problems in exchange for answers for easy problems! ;)

Most software engineers simply hate being on call and the software being developed can be pretty mundane unless you’re working on cloud native tooling perhaps. It is a rather narrow area of software engineering honestly, but IME software engineers passionate about their software in production are great SRE candidates and I say this not because I’m a former generalist software engineer either but have had to hire for these positions.

Narrow hm? In my experience as sre-swe I had to debug and write patches for kernel issues, networking issues (l3,l4 and l7), various OS issues (related to fs, cgroups, memory management), then there’s orchestration (scheduling, upgrades), safety/reliability and various configuration tooling which I had to write in Python, C++ and Go (not to mention half a dozen or so DSLs). Then there’s incident response skills for oncall.

It is much more broad than when I was an embedded dev with only one job - to make some driver work on a different architecture.

This isn’t it. DevOps aren’t the (exclusive) oncall engineers, the dev teams should be responsible for oncall as well—the people empowered to create or fix the operations problems should be responsible for operations. See my sibling comment for why DevOps is different than SE.

All true. At my current gig, the infra team is on call for pretty much everything. That's how it was when I started, and it's taken time to deal with stuff like alert fatigue and better surfacing of metrics and logs. But we're now in the process of moving to all first line pages going to the dev team (because they ship most of the bugs in the first place). If there's an infra problem, they can call us then.

If you have your infrastructure team on call instead of your developers, you are screwing up.

In almost every reasonably shaped organization the majority of bugs are shipped by developers, not infra/platform/SRE. Localize the pain to the agents who cause it or it will never go away.

Oh, not saying that’s how it should be. My current situation is such that infrastructure is the majority of the production issues and we’ll call developers on the rare occasion something serious happens relating to their code. Our platform goes through much more testing rigor than most SaaS companies our size tend to perform and I’m proud to be supporting these guys.

Probably because the role actually does still require someone to be able to put out fires while under stress. I've met many otherwise excellent software engineers who cannot or will not deal with high pressure oncall situations. The Operations aspect of DevOps requires a certain level of familiarity and comfort with the type of real-time communication and troubleshooting needed to deal with emergency situations. Some people are excellent with design and implementation work, but do not communicate well enough in high pressure situations to fit a DevOps/SRE role.

Maybe so, but my experience is that the "ops" people don't want to be "devs". As other posters have said they do want to be integrated with design choices that will affect them, and many times they do write non trivial code that keeps these complex systems on their feet.

Familiarity with different problem domains. Software engineering is essential to the DevOps skillset since we do a lot of automation, but also understanding the (constantly changing) ecosystem of tools, how to design a CI/CD pipeline, how to configure the developers’ dev environments, how to model your infrastructure as code, etc, etc. A good DevOps engineer is a good SE with an understanding of the DevOps problem space.

Because they don't want to pay software engineering wages.

The distinction is mostly because a lot of developers lack the fundamental understanding of downstack problems in 2019. It’s not that they can’t do it, it’s that they’ve rarely had to and in so not doing have built themselves a mental Jenga tower that requires time and effort to stabilize and build a foundation beneath.

Companies hate that. Investing in people who will leave is bad, they think. Put them in their box and let them do what they already know.

Which is why they hire me and call me an SRE. (I don’t use the term. My current title is “principal engineer”. I’m not an engineer, though. Neither are most people here.) And I’m not saying downstack ignorance is great. Profitable for me, sure. But it’s a natural response to companies’ unwillingness to invest in their people. They want them pre-made. Hence the made-up titles for people with breadth.

I am actively hiring for a devops. I get hundreds of resumes from sysasmins that either don't have AWS/cloud, Ansible/Automation, CI/CD lifecycle, DB management or Ngnix/Apache experience.

The majority are sysasdmins that did point and click setup ..

What types of businesses — perhaps the industries — do the majority of your applicants come from? I'm asking because, from my limited experience, it seems that many small and medium sized businesses that are NOT in the IT field are afraid of testing newer tech.

The justifications I've been given for this stance are:

1. Newer tech introduces new problems, and increases the scope of working knowledge

2. Adopting new practices requires the business to attempt to hire for that skill in the future

3. Present managers, whose experience stemmed from working in a sysadmin role, do not have the working knowledge and capability to understand/learn new practices

With that, do a lot of applicants seem to come with a basic/old background of just Windows (or the like) experience?

It's hard to try and get any of your mentioned requirements running in these businesses. What my friends and I've encountered is a big resistance to the command-line (Powershell or Bash), learning how Linux systems are configured, or anything that doesn't come with a large support contract.

If yourself or anyone on HN has tips or anecdotes on how to introduce changes — gradually and slowly AND given that it could help the business — I would LOVE to read them! My biggest goal is to reduce operating and capital costs for systems that are not accounted for contributing directly to increasing the company's revenue (at least when you don't control that calculation, anyway).

You think maybe some of those folks are looking for an opportunity to LEARN some of that tech?

The "DevOps" fad is screwing over a large segment of senior level I.T professionals who are used to specializing. (Databases, Storage, OS, Security, etc.). I've also yet to see any startup that Jez Humble would actually call a DevOps shop.

Now, startups are hiring generalists with 3-5 years of "hacking" experience, or have a popular project on GitHub.

A willingness to learn new tech is essential but not sufficient. Mostly it seems folks with a more traditional sysadmin background are looking to be Kubernetes sysadmins or AWS sysadmins, but we’re not looking for sysadmins, we’re looking for engineers. Learning the new tech isn’t sufficient—it’s not even about the tech—you need to be able to _do engineering_.

This is a critical point, and it's one the post to which you replied seems to miss.

"System administrators" in the traditional sense--and I have hired many of them and consulted on the obsolescence of others--often and generally exhibit that strong get-it-working,-damn-the-consequences tendency that is in opposition to--well, none of us in this industry are engineers, but some of us aspire to engineering. Rigorous, systemic, and repeatable are the watchwords, and to that end those system administrators aren't being "screwed over"--there's a different skillset being prioritized.

I think that is a gross mis-characterization because I see a ton of "get it done now, make it right later" bullshit among DevOps-y start ups.

Again, I think there's a large talent pool available, but startups who think they'll be the next FAANG act too big for their britches and actively discriminate against older tech workers who are likely experts in several pieces of the tech stack.

I routinely see these folks get passed over for younger, less experienced candidates (often for 1/2 to 3/4 the salary) who look good on paper because they wax eloquent about their pet project on GitHub, facial hair wax and kombucha.

Source: I make damn good money as a "fixer", and my primary customers are 5-30 person startups. I don't "code", and never will (useful scripts, and some automation/cloud API excepted).

I go in and practically beat the managers over the heads with the DevOps Handbook, and "engineers" with the NASA Systems Engineering Handbook. Most of my work is tearing out fucked k8s installs, and cutting AWS spending by 1/2 or more. (A few clients were billed based on how much I reduced their bill).

Have a standing job offer with one client, however it requires Azure certification pretty much immediately. Between not really using much MS stuff, and the exam focusing mostly on the Azure CLI, it might not be worth the trouble for a steady paycheck. They were nice enough to cover a training course though, so I'm willing to see where it goes.

Sure, there are stupid startups that think they do "devops". What of it?

It's great that you can make that money in the role you describe. Before I decided I wanted to stop doing sales work alongside dev work, I used to make very good money as a similar fixer. On the other hand, I do code. I'm very good at it. And I've learned that fixing the situations of companies whose operators don't code is to fix, or replace, those operators. Especially those expensive operators who you're holding up over folks who understand systems as code and as managed resources.

Sneering at kombucha and facial hair wax, though? Aren't you saying you're the adult in the room here? Frankly, you sound bitter. And that sucks. As somebody who has spent his entire career doing both dev and ops and getting to the point where melding them together is natural and the teaching thereof is likewise a basic part of work, I've had to recommend the replacement of people who act like you're acting in this thread. 'Cause I'm happy to teach, and I've never met a hands-first sysadmin who couldn't do what should be done. But I've met a lot who won't, and if they don't retire first it eventually catches up to them.

So you are part of the problem then. You've globbed onto a bastardized concept of "DevOps", and refuse to allow other people into your walled garden.

There is no authoritative definition of DevOps, so there’s really no point in arguing that one of our definitions is wrong. I’m telling you that it is more valuable to treat ops as an engineering problem rather than “duct-tape it and keep it chugging along”. So yes, if you want to treat it as an engineering problem, you must _employ engineers_. These engineers can be former sysadmins so long as they know how (or can be trained in a reasonable timeframe) to do engineering.

WRT your “walled garden” quip, employment is about qualifications. No one is entitled to jobs for which they are not qualified, not even sysadmins. If the employer is hiring engineers and the sysadmin candidate can’t or won’t learn how to engineer, then they are not qualified.

Hi! I wanted to say a couple small things in followup to your recent submission about coding, but that thread has now locked and is not accepting replies (<rant>my biggest dislike of HN</rant>). Really glad you're still posting comments and I can reply to this one! My email is in my profile (click my username).

Yeah, I’m not complaining about the state of the industry - it makes sense why you want people who can code! My advice is more for people in DevOps who can code that because of this inundation of sysadmin applicants who can’t code, you need to make sure that you are distinguishable from the herd so that your application doesn’t get rejected immediately based on your title. That’s why the title really matters, and also why it’s important to actually write code as a DevOps. Any role where you don’t get to code is career-limiting, IMO.

SREs make 5-10% more than SWEs at Google.

That’s good to know, thanks for bringing that to my attention. Is it true that SREs are more likely to have oncall rotations at Google than SWEs? That is the impression I got from reading the Google SRE book - it talks about bringing SWEs into the oncall rotation when the SREs are swamped (i.e. more than 50% of their time is devoted to operational work) as a kind of pressure release mechanism to prevent the SREs from burning out or getting mired in endless toil. It makes it sound like outside of situations like that, SREs are doing most of the oncall work.

If this is the case, perhaps the pay discrepancy is explained by the greater oncall duties and morale issues for SREs?

It’s also worth mentioning that based on my reading of the Google SRE book, very few organizations approach operations like Google does. I personally think that Google has an enlightened approach to operations, but not all companies do.

Basically, I think SRE is a good title to have, but sysadmin is not, and I encourage anyone early in their career to rebrand themselves ASAP.

> I personally think that Google has an enlightened approach to operations, but not all companies do

Perhaps because the founders were doing operations (well) in the beginning, so they know a few things about it and its importance. They famously built a reliable system with about the cheapest possible hardware.

Are SWEs at Google expected to firefight especially outside of normal work hours? This is one thing I imagine SREs are expected to be doing that regular SWEs aren't. I think this is also why they require strong development background for the role.

I worked in Amazon which has a similar view to on-call to Google. Usually one person is on-call for the team for a cycle of every week or fortnight. There is also a "follow the sun" model wherein another team in has your nights covered. I do recall that SWEs also had the responsibility of being on-call, since teams are usually a mix of SWEs and SREs.

Naturally a team with a high ops load was shit and made your life hell. After my time there, I vowed never to do on-call unless it was after hours and specifically for emergency response. My opinion is - if there's an issue during work hours, the leads or the entire team should be on it - and collectively fix it.

Anyway, Google has written a bit about this in their SRE book: https://landing.google.com/sre/sre-book/chapters/being-on-ca...

Which kind? There are SRE that are SWEs and then there are plain SREs.

Reply on news.ycombinator.com