In retrospect, DevOps was a bad idea
rethinkingsoftware.substack.com
rethinkingsoftware.substack.com
The idea of a single entity being responsible for development, operations, observability, and support is flawed from the start. That’s not a one-person job, and the expectation simply doesn’t scale. So DevOps often ends up being either ops folks or dev folks, and rarely a true blend of the two.
What we need are feature-focused developers, ops-savvy devs who can deploy their own work, and a strong team dedicated to observability and applying modern SRE practices.
So I think curious developers who aren’t afraid of infra, along with a solid platform engineering team, are a real improvement over the status quo.
The expectations were (and are) irrealistic.
Looking for somebody that has both the software engineering skills of a software engineer and the skills of a sysadmin, willing to do two jobs for the salary of one: yeah sure, keep looking for it.
That being said, i'm one of those that jumped on the bandwagon because in practice it meant fancier job title, higher pay and getting to use newer (often better) tooling.
Ten years ago being a devops engineer rather than a sysadmin usually meant getting to work with EC2 and cloud stuff rather than administering remote physical servers and fighting with NetApp SANs.
Devops was meant to be really a methodology change, not really a job title.
In my experience whether DevOps works or not really depends on the management. I can setup all the automation you want, but management must back me when I tell you're supposed to use the automations rather than involve me into doing your work.
I can surely take feedback and improve whatever you need to be improved, but you *must* be using the automation.
Otherwise we go back and Development and Operations.
Why? I have seen this done successfully at my work time and again. I've just switched to a company that has this separated out and the chaos is unimaginable. what's even worse is that everything is a finger-pointing exercise. devs can't ship their features because they have to fight (or wait) for DevOps to set up things. SRE is trying to add observability and is swamped with 18 teams having services in different tech stacks. Service health is basically non-existent. If someone tries to set a standardized process, then 'how dare you suggest anything outside of your domain?'. Everyone seems to focus on what they think is right and are rarely convinced by others. Worst, not even a tiny bit taking seniority into consideration. not saying seniors know it all better, but the one or two things they have seen are fully disregarded.
At FAANG at least, each team has full ownership AND responsibility across all dimensions. But the good thing was that the people knew that and if anything went wrong, then business was asking hard questions and service teams (product, ux, engineer, sometimes science) had to explain and potentially pivot their processes. which was easy, because everyone was pulling on one string together.
> That’s not a one-person job Correct, that's why you are working as a team on all these things together.
The dev in me hated throwing an app over the wall, only to hear about ops completely misconfiguring it. Or ops ignoring deployment guidelines and mucking up releases. Or getting a db with 2 records and being expected to test performance. Or getting complaints about prod crashing, but no logs, eror messages, or any kind of access, and ops couldn't notice that disk full might be a disk being full.
The ops in me hated devs being loose with passwords, asking huge mountains of resources, and getting apps thrown over the wall at me without any idea what to expect. Or devs logging sensitive stuff.
For me, devops was good for breaking up the silos. We wanted to have a group of people cooperating on a group of applications. Some people will have a dev or ops profile, and that's fine. People should respect that others know more of some aspects. But: no one should shield the applications from everybody else in such team. Everybody should strive to learn enough basics to understand both perspectives and priorities.
Starting a devops silo is just the worst possible option.
Take any combination of dev, platform engineering, devops, SRE, ops teams (existing or not) and combine with different responsibilities and how they work together and there's a company out there using that model.
Heck, we don't even have a common understanding of what a "DevOps" team does https://docs.sadservers.com/blog/what-the-f-is-devops/
Very good point, that's the same I have observed for the past couple of years when working on a devops team. Product team engineers nowadays feels like spoiled kids, they had no current how server runs, and asked for things unreasonable. I still remembered someone came to my desk and asked for me to increase the mem request to 10s of GB, he claimed that's the best solution he could think of is to load everything in mem.. and very often people don't even know what status code means 500, 502, 503, 504...
But those people are making $300k+ at FAANG companies.
Unfortunately they are also the ones who write all the blog posts and podcasts on devops
For my tone, I apologize, but there is just so much wrong in this post that I don't want to go through it all, but here's one example of why I think this person, with all due respect, has very little idea of what they are talking about. It's irritating because it seems to devolve into some weird rant about "locked doors" which are 100% and absolutely necessary in any operation of size and scale:
> Originally, DevOps was about trusting developers with production. But modern DevOps teams operate on the belief that developers can’t be trusted with production.
I don't even know where to begin with this one. Who hurt this guy?
> And because DevOps owns the compliance checklists, they bake that mistrust into the rules.
That isn't remotely how compliance works and suggests a child-like understanding about any of this. At most they are responsible for implementing the checklists, which comes from a security team, compliance team, or some external entity, and even those checklists are not really generated or controlled by these entities, they are following established guidelines already in place that most people need to adhere to. So, red flag number one this person doesn't know what they're really talking about, but I'll digress to my main point because I think the spirit of the post is correct even though most everything in it is silly.
I have the opinion that the industry fundamentally misunderstood what DevOps was supposed to be and do. Of course this is rarely challenged, but where these arguments fall short are what are we supposed to do instead? The author comes close to recommending a solution - have them work closer to the devs and thus the product teams. Boom. Simple. That's it, and in fact, tons of places not stuck in the practices of 20 years ago have done this, and successfully! They're just not typically called "devops." Like, at meta for instance, the closest thing to that (and forgive me if I'm misremembering or this has changed, this is from a round of interviews some years ag) is a role called "systems engineer" and they work on and with the dev teams.
The existence of a team called "DevOps" to me is a massive red flag anymore that a company isn't getting it, but I have still seen this "chuck it over the wall at each other and pray" approach work fine enough. It's also created a job title that can mean practically anything and predicts very little about a person's skill set - you have "senior" devops guys and leads that basically got promoted from a Sys admin or IT support role, which for some reason companies thought was a perfectly natural progression, and then devs who did actual development for a while and now write fully functional platforms and automation for the dev teams they support. These people are not the same, which makes the hiring/interviewing process a nightmare. To make it worse, even tech people can barely understand the difference sometimes. So you'll run the gamut of having teams full of "devops" guys that can barely string together 6 functional lines of code, to full stack guys that could probably work with any of the dev teams they support as a dev.
I don't pretend to know the fix but I've increasingly gone out of my way over the last several years to avoid "devops" titles because it just seems to be a dead giveaway that the team is clueless or at best apathetic about how to do this.
This right here pretty much discredits him (or her) completely; there's no way you would survive five minutes of an audit where developers have direct access to Prod in any kind of regulated business (finance, medical, banking, nuclear, military etc). This might be acceptable for an owner operated website that sells trading cards but absolutely not for any company > 500 employees.
I worked for a bank for a short period of time, where the development team had gone a bit far with a shotgun approach to moving their systems to Azure. It was pretty hard to find things and as their approach evolved, the older conversions weren't revisited. Most of that team quit. New team is brought in (including me) which had some excellent engineers with years of experience in system architecture, how to make Azure work better. So we tried to homogenise things around a more reasonable approach.
This caused operational problems to worsen (understandably) but it was a short term pain, long term gain thing we couldn't be allowed time to do. So the CIO decided to take all the sysadmin/devops type work and give it exclusively to the system administrators. Who weren't developers. They fixated on one particularly narrow solution for deployment. To make it easier for themselves, but one that didn't really address the bigger picture of how to make it easy to monitor and deploy etc.
Anyway it ended up a disaster. The development team in their newly narrowed roles struggled to make their systems fit in the rigidly defined holes. Operationally it was no better and sometimes worse, but there was absolutely no compromise on how things should work or any consultation with the devs at all, ever.
I no longer work there. If you're going to do devops you have to listen to your experienced engineers, not the snotty kids who think clickops is engineering.
But… I’m going to say the dirty, quiet, and unlikable thing out loud.
That had nothing to do with DevOps or its philosophies, processes, or patterns. That was bad leadership from the top down plain and simple. It’s likely not even the individual engineers faults. It’s leaderships fault for not setting clear objectives, implementing them, ensuring that the engineers had a real plan before beginning, and making sure no individual was too in charge of things.
Leadership in your case was likely career management who knew very little about technical items. Managers who were technical were probably shot down for not playing politics properly, not producing the correct “metrics” and “kpis”. So they moved on.
That’s a company culture issue that has little to do with tech.
I have known and worked with some really great former sys admins gone devops. I am working on mentoring one right now, but I have to be kind of insulting about it and be like “forget everything you knew before it probably won’t help now” which sucks because sys admins do form pretty decent understanding of OS’s, databases, networking, etc. however, when it comes to the code part and more importantly taking all of these concepts and applying them to reasoning about infrastructure code and complex systems is very hard for most people and you have to take a “im a total newb” mentality a lot of people dont seem easily capable of doing.
Still, it made me very wary of the idea that devops is separate to development.
The complete, unapologetic desire of devs and security teams (but also many infra teams) to not have any kind of ownership was horrifying to me.
In the end there's not a single solution or strategy, it really goes back to the organization and where your weaknesses and strength are as an org. If you have a gazillion consultants following the "best practice" of the day and exceptions on top of exceptions you are dead, devops or otherwise. You will still make billions if you are the right company though regardless of your software practices, so...