Oh I like that one. "I kept the pile of shit running and put the fires out before they were noticed" is too long to put on a business card, anyway. "Site Reliability Engineer" sounds almost tame.
I've been at my current company for 6 years now. When I first started, those guys were "Operations", and the job title was "Admin". We used Google App Engine, and they mostly clicked buttons in the GCP web console.
One year in, they re-branded as "DevOps". They were all "DevOps Engineers". They wrote a lot of hacky Python scripts, using the "gcloud" utility, to manage our new Docker-based services on Google VM's.
A year or two later, they decided to become "Platform Services". They changed their titles to "Platform Engineers". They saw us developers using Jenkins for our CI/CD pipeline, and decided to re-implement all of their Python scripts as Jenkins jobs.
Earlier this year, they became "The SRE Group". They are "Site Reliability Engineers" now. We eliminated their Python scripts, by migrating our microservices to Kubernetes and using managed databases. So now they're back to clicking buttons in the Google Cloud Platform web console again.
So much for DevOps philosophy.
So much for DevOps philosophy, indeed.
Any real, experienced DevOps engineer will reply “a steaming pile of shit”.
You will need to maintain separate TF codebases for each one anyway, but with the native solutions you get easy access to all features of the platform and don't have to spend most of your time jumping through stupid hoops to try and pretend that one tool can do everything.
No-one would write code worrying if it was valid in both Java and C# but that is the same level as what TF claims to do. It's complete shit, ditch it and you will be 10x happier, I guarantee it.
Boto doesn’t seem comparable to terraform at all, unless you enjoy building state management yourself.
Terraform has been better than anything else I’ve tried so far.
Now, it is also true that most developers are just content with clicking around in their Jetbrains IDE without bothering about resources or even knowing what a file descriptor or system call is. Those so called senior Java engineers that I had to explain the basics of garbage collection to them. Those also make operations harder than it should be.
But that’s another story. We’re taking about the dissonance between what gets advertised as DevOps and what truly is, which seems to be the norm in our industry and is leaving a significantly amount of people unhappy and dissatisfied.
Nonsense. They're smart professionals, and probably the hardest workers in the company. Certainly the ones subject to the highest degree of pressure, which they always handle with grace.
I "speak" of the absurdity of our industry's superficial hype cycles, and forever-swinging pendulums. After you've been in the game for 20+ years, you will recognize a number of cycles and pendulums yourself. You'll go along with it, because that's part of being a professional. But you'll wonder why we're collectively unable to step back, and recognize more of these things, and not pointlessly churn so much.
However, I could write a similar comment about engineering managers and SDLC methodology trends. A similar comment about testers and quality assurance. And yes, probably a dozen similar comments about language fads and architecture patterns and trends related to developers.
I don't think any of those comments would come from a place of disdain for workers themselves. I just think that culture has shifted, and anything short of unqualified gushing praise registers as offense for many younger people. Maybe that pendulum will swing again too?
They seem to have bought into their own myths.
I am a software developer and yet when I hear a developer open their mouth to complain about nearly any other aspect of a product team, be it QA or infra or UX, it has this weird tendency to resolve to Anybody But The Developer causing an issue.
That said, they are highly paid, London financial world folks, so you'd hope they'd be good.
I also hear from a developer I work with who has recently defected back from devops style roles to devs that this is not atypical. He seems to place blame at least partially on "The Phoenix Project" for the attitude!
At least I had root access to my own workstation (and everybody else's in the company, thanks to the fact that NFS actually stood for No File Security).
[In the late 80's and early 90's, NFSv2 clients could change their hostname to anything they wanted before doing a mount ("hostname foobar; mount server:/foobar /mnt ; hostname original"), and that name would be sent in the mount request, and the server trusted the name the client claimed to be without checking it against the ip address, then looked it up in /etc/exports, and happily returned a file handle.
If the NFS server or any of its clients were on your local network, you could snoop file handles by putting your ethernet card into promiscuous mode.
And of course NFS servers often ran TFTP servers by default (for booting diskless clients), so you could usually read an NFS server's /etc/exports file to find out what client hostnames it allowed, then change your hostname to one of those before mounting any remote file system you wanted from the NFS server.
And yes, TFTP and NFS and this security hole you could drive the space shuttle through worked just fine over the internet, not just the local area network.]
What that actually did was change the "DEBUG" command into the "" command.
At the time I was running a mailing list from the University of Maryland, and often had to check Sun email addresses by telnetting to sun.com port 25, pressing return a couple of times to flush the telnet negotiation characters, then going "EXPN some-email-address".
So the day after the Morris worm, I go "telnet sun.com 25", hit return a couple of times, then "EXPN foobar", and it dumps out a huge torrent of debugging information, because I had accidentally switched it into debug mode by entering a blank line!
I reported it to postmaster@sun.com, and they fixed it. But it's kinda silly that they would have applied such a ham fisted patch to their sendmail demon like that, based on an email that some dude on the internet sent around!
https://www.rapid7.com/db/modules/exploit/unix/smtp/morris_s...
http://www.cs.unc.edu/~jeffay/courses/nidsS05/attacks/seely-...
I have the job title of SRE currently and could care less what the actual title is. I've got work to do, and I'm going to accomplish it in the best way possible given whatever constraints exist at that time.
Switching my title from "System Administrator" to "SRE" within my last company resulted in a job family change and a 10% raise (I had to show I could code as well as a software engineer in order to make the switch).
When I left that role, having "SRE" on my resume instead of "sysadmin" was (probably, I don't have any strong evidence for this) instrumental in getting responses when I applied to "Software Engineer" roles at selective companies.
I think there is a bias against roles that don't code, and especially roles that sound outdated.
A lot of larger companies and managers in those companies don't understand what DevOps is, what SRE means or anything like that. They just know Ops and Dev, and your previous job title is probably the strongest hint they have to work off when they categorize you. Getting lumped in with Ops is (probably) a big hit to your earnings potential and limits your future options if/when you decide to move.
Perhaps we're in agreement, but this is critically important. When my company hires for DevOps, CloudOps, etc positions, we are innundated with applications from Ops/SysAdmin personas. We don't want people who will put out the fire and keep the system limping along--we want people that can't stand putting out fires _and who have the skill set_ to build systems that (1) aren't likely to catch fire and (2) are easy to troubleshoot/extinguish when they do. One way we're addressing the problem is to change titles from "DevOps" and "CloudOps" to "Cloud Engineer"--not sure yet how big of an impact that will have (if any at all), but it's worth a shot.
The majority are sysasdmins that did point and click setup ..
The "DevOps" fad is screwing over a large segment of senior level I.T professionals who are used to specializing. (Databases, Storage, OS, Security, etc.). I've also yet to see any startup that Jez Humble would actually call a DevOps shop.
Now, startups are hiring generalists with 3-5 years of "hacking" experience, or have a popular project on GitHub.
"System administrators" in the traditional sense--and I have hired many of them and consulted on the obsolescence of others--often and generally exhibit that strong get-it-working,-damn-the-consequences tendency that is in opposition to--well, none of us in this industry are engineers, but some of us aspire to engineering. Rigorous, systemic, and repeatable are the watchwords, and to that end those system administrators aren't being "screwed over"--there's a different skillset being prioritized.
Again, I think there's a large talent pool available, but startups who think they'll be the next FAANG act too big for their britches and actively discriminate against older tech workers who are likely experts in several pieces of the tech stack.
I routinely see these folks get passed over for younger, less experienced candidates (often for 1/2 to 3/4 the salary) who look good on paper because they wax eloquent about their pet project on GitHub, facial hair wax and kombucha.
Source: I make damn good money as a "fixer", and my primary customers are 5-30 person startups. I don't "code", and never will (useful scripts, and some automation/cloud API excepted).
I go in and practically beat the managers over the heads with the DevOps Handbook, and "engineers" with the NASA Systems Engineering Handbook. Most of my work is tearing out fucked k8s installs, and cutting AWS spending by 1/2 or more. (A few clients were billed based on how much I reduced their bill).
Have a standing job offer with one client, however it requires Azure certification pretty much immediately. Between not really using much MS stuff, and the exam focusing mostly on the Azure CLI, it might not be worth the trouble for a steady paycheck. They were nice enough to cover a training course though, so I'm willing to see where it goes.
It's great that you can make that money in the role you describe. Before I decided I wanted to stop doing sales work alongside dev work, I used to make very good money as a similar fixer. On the other hand, I do code. I'm very good at it. And I've learned that fixing the situations of companies whose operators don't code is to fix, or replace, those operators. Especially those expensive operators who you're holding up over folks who understand systems as code and as managed resources.
Sneering at kombucha and facial hair wax, though? Aren't you saying you're the adult in the room here? Frankly, you sound bitter. And that sucks. As somebody who has spent his entire career doing both dev and ops and getting to the point where melding them together is natural and the teaching thereof is likewise a basic part of work, I've had to recommend the replacement of people who act like you're acting in this thread. 'Cause I'm happy to teach, and I've never met a hands-first sysadmin who couldn't do what should be done. But I've met a lot who won't, and if they don't retire first it eventually catches up to them.
WRT your “walled garden” quip, employment is about qualifications. No one is entitled to jobs for which they are not qualified, not even sysadmins. If the employer is hiring engineers and the sysadmin candidate can’t or won’t learn how to engineer, then they are not qualified.
The justifications I've been given for this stance are:
1. Newer tech introduces new problems, and increases the scope of working knowledge
2. Adopting new practices requires the business to attempt to hire for that skill in the future
3. Present managers, whose experience stemmed from working in a sysadmin role, do not have the working knowledge and capability to understand/learn new practices
With that, do a lot of applicants seem to come with a basic/old background of just Windows (or the like) experience?
It's hard to try and get any of your mentioned requirements running in these businesses. What my friends and I've encountered is a big resistance to the command-line (Powershell or Bash), learning how Linux systems are configured, or anything that doesn't come with a large support contract.
If yourself or anyone on HN has tips or anecdotes on how to introduce changes — gradually and slowly AND given that it could help the business — I would LOVE to read them! My biggest goal is to reduce operating and capital costs for systems that are not accounted for contributing directly to increasing the company's revenue (at least when you don't control that calculation, anyway).
It is much more broad than when I was an embedded dev with only one job - to make some driver work on a different architecture.
In almost every reasonably shaped organization the majority of bugs are shipped by developers, not infra/platform/SRE. Localize the pain to the agents who cause it or it will never go away.
A SWE typically won't want to touch infra. That's my experience of course, YMMV.
I have fond memories of the team that worked across three time zones so I would get up to a set of well-described problems that I would solve in the morning (so satisfying), then a nice stretch of feature building after lunch, then a burst of pair coding with my newly arisen colleague, then maybe finishing with writing up any roadblocks or requests for the next person. I got very used to identifying blockers that were out of my area and would be better solved by the domain expert, and also a LOT better at ticket writing. It was a really, really productive and rewarding workflow and one of the key points was not getting bogged down in stuff outside my areas. We also all really appreciated each other because we all experienced each other as magical elves giving answers to hard problems in exchange for answers for easy problems! ;)
Companies hate that. Investing in people who will leave is bad, they think. Put them in their box and let them do what they already know.
Which is why they hire me and call me an SRE. (I don’t use the term. My current title is “principal engineer”. I’m not an engineer, though. Neither are most people here.) And I’m not saying downstack ignorance is great. Profitable for me, sure. But it’s a natural response to companies’ unwillingness to invest in their people. They want them pre-made. Hence the made-up titles for people with breadth.
If this is the case, perhaps the pay discrepancy is explained by the greater oncall duties and morale issues for SREs?
It’s also worth mentioning that based on my reading of the Google SRE book, very few organizations approach operations like Google does. I personally think that Google has an enlightened approach to operations, but not all companies do.
Basically, I think SRE is a good title to have, but sysadmin is not, and I encourage anyone early in their career to rebrand themselves ASAP.
Perhaps because the founders were doing operations (well) in the beginning, so they know a few things about it and its importance. They famously built a reliable system with about the cheapest possible hardware.
Naturally a team with a high ops load was shit and made your life hell. After my time there, I vowed never to do on-call unless it was after hours and specifically for emergency response. My opinion is - if there's an issue during work hours, the leads or the entire team should be on it - and collectively fix it.
Anyway, Google has written a bit about this in their SRE book: https://landing.google.com/sre/sre-book/chapters/being-on-ca...
Change your title to DevOps, SRE, or team lead. I also don't like titles, I'm sharing my experience with you.
In Viscera Cleanup Detail, players are given the role of "Space-Station Janitors", tasked with cleaning and repairing facilities that have been the scene of bloody battles during an alien invasion or other form of disaster. Tasks include gathering and disposing of debris, including, dismembered bodies of aliens and humans, spent shell casings and broken glass, restocking of wall-mounted first-aid kits, repairing bullet holes in walls, and cleaning of blood splatter and soot marks from floors, walls and ceilings, as well as secondary bonus tasks. These include stacking items like crates and barrels in a designated stacking area, and filing disaster reports on the events and deaths that took place in a corresponding level.
https://en.wikipedia.org/wiki/Viscera_Cleanup_Detail
Writing for The A.V. Club, Chaz Evans called it a commentary on first-person shooters that focuses on the consequences of violence.
https://games.avclub.com/seeing-without-crosshairs-a-survey-...