Hacker News Reader

Comment by saurik | Hacker News Reader

> If you look at a DevOps engineer job description, it looks remarkably similar to a System Administrator role from 2013, but...

> If DevOps was supposed to be about changing the overall culture, it can’t be seen as a successful movement. People on the operations side of the fence will...

As someone who was keenly watching this stuff back 15 years ago, parts of this article connect with my understanding, but the core problem I have is that this article itself is somehow bought into the mistake that led to the failure and so almost can't see the failure for what it is: the entire point of DevOps was that "operations" isn't a job or role anymore and has instead become a task that should be done by the developers.

Ergo, if you even still have operations people to comment on it--or certainly if you are somehow hiring dedicated "DevOps" people--you aren't doing DevOps and have already failed. The way to do DevOps is to fire all of the Ops and then tell all of the Devs that they are now doing DevOps; you simply can't have it both ways, as that's just renaming the same two camps instead of merging them into a single unified group.

View on HN

I've worked in ops roles since about 2000 after a few years in backend corp IT stuff.

I agree that what you've described was the original intent and goal of "devops" but in light of that failure, the "cross functional team" definition took over and then in light of that failure, the SRE was born and we're basically back where we started but now the ops people use git instead of rcs.

In my experience and opinion, developers are really bad at ops and sysadmins/ops are really bad a development. Anyone that is truly good at both is a unicorn that is probably carrying their team.

Hey, this comment inspired me to create a pool. I'd like to know what is the distribution of unicorns between HN crowd :)

Pool: https://news.ycombinator.com/item?id=31891675

Why not sit down and figure out how we can train more unicorns instead.

And maybe make the tooling helps at this

because it's not "profitable", it just doesn't make sense for 95+% of teams.

most teams are not building the next facebook. most teams have at least some stability, most teams benefit from delegating specific tasks to specialists (eg at project kickoff they talk to the various leads they sketch out a design, agree, and get back to their respective terf, and when it comes time to deploy it they again talk to whoever and in a few iterations it gets deployed, it goes into testing and then into production, and that's it)

sure, there's always bitching about how it's not agile, but ... like I said, they don't have next-facebook-like money.

maybe Netflix is the best example for this. everyone was in awe of them for how they are going all in with Cassandra on AWS, microservices, flamegraphs, circuit breakers, 30% of the Internet traffic, and many groups/startups started copying them. but forgot a few tiny details like paying half a million dollars a year to new recruits and having a cashflow that can sustain all the aforementioned things.

almost every group would benefit from a more holistic knowledge of whatever they are doing as a whole. but just as there seems to be a natural limit to how many peers one can comfortably have at the same time (eg Dunbar's number) it seems people naturally like to set up softer or harder boundaries for their IT knowledge. ¯\_(ツ)_/¯

... and yes, tooling seems to be the place where this kind of complexity should live, but then maintaining that tool becomes the real challenge :)

I would argue that it is profitable, as it is far less expensive than the current shitshow. In particular for smaller companies. I have seen team of "unicorns" and you can do the equivalent of 100 devs organisation with a couple team. WhatsApp or Discord come to mind.

Also that tooling could be opensourced and shared you know :D It does not have to be a price paid by everyone building their owns.

But it would indeed have to handle the real problem. Not like the current tooling that our ops people force down everyone throat :cough: K8s :cough:

> the entire point of DevOps was that "operations" shouldn't exist, and that operations is a task that should be done by the developers. Ergo, if you even have operations people to comment on it, or if you are hiring dedicated "DevOps" people, you aren't doing DevOps and have already failed.

This. My first thought when I was reading the article. Spot on

> the entire point of DevOps was that "operations" isn't a job or role anymore and has instead become a task that should be done by the developers.

This akin to saying "frontent developer isn't a role anymore - both frontend and backend should be handled by a full-stack developer". This works for small companies/projects, but bigger ones can benefit from specialization and division of labor. Body of knowledge required to be a decent software developer and a decent ops engineer is too big to fit into one head. I've seen ops work being done by developers without ops experience and more often than not it was ugly - they didn't had enough experience/knowledge (and time/incentives to gain them) to do ops work well.

To me the best part of DevOps isn't about roles but about team structure - splitting all Dev in one department and all Ops into another usually is a bad idea. And a failure of this split was a motivation to start DevOps movement. Having Ops embedded into Dev teams in my experience works much better.

> but bigger ones can benefit from specialization and division of labor

I think it is central to the DevOps concept is that dev vs. ops segregation—at least on the small team level, perhaps not at the individual level—is a counterproductive division of labor that inherently fosters micro-optimizations on both sides of the divide that are counterproductive to effective value delivery. On a continuously available software service, the lowest-level product team should own it's components soup to nuts rather than having a dev team throwing hopefully-deployable code over the wall to an ops team.

And that's great until you have three-ish "lowest-level" product teams and they're all managing not only their own production systems, but their own deployments and testing pipelines and monitoring stacks and secrets management. At some point, you need to start unifying that stuff to keep things manageable, and if those systems are everyone's responsibility, they're no one's. So you make another team for whom the internal shared infrastructure is the product, and the rest of engineering are the customers - they still write their own Terraform modules and whatnot, but they run them on DevOps-supplied platforms. That seems to be what modern DevOps is becoming.

these are not mutually exclusive. if those product teams are under one big umbrella then nothing stops the one holding the umbrella (C-level or whoever) to set the standard. you can use these languages, these CI tools, these package repositories, etc. if you need something exceptional ask.

the important aspect is that there should be enough working knowledge about these tools/processes/systems in those teams that they can work efficiently, and they can respond as the whole business evolves. (eg. scale up/down, extract and hand over or accept and integrate components, integrate other APIs, etc)

> and if those systems are everyone's responsibility, they're no one's.

again, there's a cut off eventually. for example many companies just use GitLab for CI. at that point it's up to the big umbrella to decide whether they want to be in one or many GL orgs.

> That seems to be what modern DevOps is becoming.

sure, but that cutoff seems to be infrastructure vs product. which seems a bit more healthier than cutting the software lifecycle in half.

One can have both - team(s) which own shared infra like k8s, monitoring servers, CI/CD servers e. t. c. and ops embedded into development teams.

> Body of knowledge required to be a decent software developer and a decent ops engineer is too big to fit into one head.

the usual answer is that in a good agile team everyone should be a little bit T-shaped

https://www.cybermedian.com/scrum-team-i-shaped-vs-t-shaped-...

of course domain experts are real (or what's the term nowadays?), so specialization makes sense. comparative advantage and all. but the idea is to lower the (coordination, communication, conflict due to inevitable misalignment between separate teams) overhead by "onshoring" the basics (eg. writing tests, basic CI stuff, deploying)

the devops manifesto (which allegedly does not exists, but you get the point) basically calls for giving people tools, permissions and authority to do these basic things, giving teams ownership of their stuff. and of course this doesn't mean fire every sysadmin on sight :D (even if that would definitely help with the process of re-owning some ops taks to dev people)

> The way to do DevOps is to fire all of the Ops and then tell all of the Devs that they are now doing DevOps

That's like saying "agile is firing your scrum masters and tell your developers you're agile now".

The idea behind DevOps is that applications are provisioned by the people who know the application best, the developers. If everything works out as it should, this gives additional load with creating the deployments, but also removes the overhead when dealing with operations when deploying, updating and debugging - so a net zero in workload, but a gain in the way the application is hosted better and fixing bugs is easier. You still need operations, both for providing the underlying platform (getting a server ready is not a developers core business and it shouldn't be) and for guiding the developers. It should be leaner, but you still need it.

Of course, you can also fire all of infra and tell the developers "that's your job now", but that's like calling biweekly deadlines scrum (and leads to equally bad outcomes).

> That's like saying "agile is firing your scrum masters and tell your developers you're agile now".

This would be an excellent idea in many teams.

Yes - and it's also true for many DevOps people. Some would argue that's even true for many managers. It's just not a good idea if you want to get where these people where supposed to get you when you hired them.

I’d say that is absolutely untrue in the case of agile.

Those promoting scrum (and especially those with certifications in it) are going to lead to scrum, not an agile process that values producing useful, working software each iteration.

This can be true, but I would argue not always. Some DevOps teams work in the old mode of “throwing code over to Ops to run” - this isn’t what DevOps intended, but happens.

When they work well, they’re doing things like authoring reusable (by product eng. teams) infrastructure modules, or helping to build “you build it, you run it” tooling like monitoring stacks etc. They’re also helpfully/hopefully subject matter experts on CI/CD, your cloud/hosting of choice, security stuff - things that general developers have mixed levels of interest or competence in.

That is utter BS. DevOps means Ops and Devs working hand in hand in a crossfunctional team. Nothing more, nothing less. The main idea was to tear down silos.

Well, you can't simultaneously tear down silos and continue to have two silos... I would hope that would be obvious? The new cross-functional team is made up of the Dev people who are now doing Ops and the Ops people who are now doing Dev, with everyone else no longer fitting into the new DevOps reality, which was itself born from the premise that cloud computing was simply obsoleting the floor of dedicated systems administrators you previously had building machines and coordinating workloads and replacing it with a new deployment paradigm where a developer could develop their operations as easily as as they can develop anywhere else in the product's stack. If you have a special DevOps team you hire people into that is simply a renaming of the people you previously had doing Ops, you either haven't internalized this future or are actively rejecting it (which, I will emphatically state, is a perfectly fair position to maintain) and of course are going to "fail" at DevOps.

> Well, you can't simultaneously tear down silos and continue to have two silos... I would hope that would be obvious? The new cross-functional team is made up of the Dev people who are now doing Ops and the Ops people who are now doing Dev, with everyone else no longer fitting into the new DevOps reality,

There's a project I'm tangentially involved in, that has app tiers written in different languages where the devs in either tier don't actually know the language of the other tier. They're still on the same team; they talk at the morning meeting, they work together to get things done, they have the same overall set of goals.

Despite not doing eachother's work (or even being able to), they're not in silos. There's no throwing stuff over the wall.

> DevOps means Ops and Devs working hand in hand in a crossfunctional team.

Not sure how typical is my experience, but in my experience it always meant "ops is yet another task for the Devs".

Seems like this is a good interview question: "What does 'DevOps' specifically mean in your company?"

Coming from the other direction, I've hired for DevOps roles a couple times and as a hiring manager one of my very first first questions to a candidate is to ask them to define DevOps, then ask them to define their previous role(s) against that ideal because DevOps is so broadly used as a title.

It's a great question because it finds misalignment rapidly. Post a job for DevOps and you'll get everything from old-school sysadmins to software developers of all flavors to AWS Certified Somethings applying.

> That is utter BS.

This is not conducive to the desired environment.

On a more even note I would prefer you spend some time looking at the origins of devops and what it means, it’s a contentious term because it means different things to different people.

The original “Patrick Dubois” (founder of the term) meaning was Systems Administration in an agile fashion.

I suspect that you’re repeating what someone else told you and you’ve just adopted their definition, which is fine, but part of the issue I have with the term myself is that everyone has another meaning than everyone else.

At all the places I worked previously in the last ~20 years there was always a sharp separation of development/testing and production environments. I as a developer never had access to any system in production, apart from one place which had a very sophisticated security system in place which could grant you temporary access during deployments. Just think about customer data, and you'll understand why.

So when I hear that someone thinks devops is developers running their own systems in production I always wonder where this is actually possible, let alone whether it is a good idea at all.

I have the opposite experience.

Given that I've been perfectly capable of doing ops work in the previous 5 companies I've worked at, suddenly being unable to do so in my current company because I'm classified as a 'dev', is supremely frustrating.

Especially when you have more experience by yourself than the entire ops team combined.

You're just replacing one set of people with access to customer data with another.

In either case you should be implementing least-privileges. Only the access to data that a person needs to get their job done. More frequently this is developers than operations people.

> The way to do DevOps is to fire all of the Ops and then tell all of the Devs that they are now doing DevOps

That's a way to say you're doing DevOps, but it's not going to work very well.

> merging them into a single unified group

That's the right way to describe it. Just like the old "programmers build it to spec, then throw it over the wall to QA" is out of style, and now good teams have testing specialists in the same room (conceptually) as developers. the goal with DevOps was to stop doing the old "here's a build, deploy it" that caused so much wailing and gnashing of teeth and instead bring ops skills into the team as a first-class expertise. The CI/CD pipeline can mean that development, testing, and release are all together, rapidly iterating and responding to change.

By the way, pretty soon the AppSec/CyberSec people will be folded in, to, so instead of the old "it's done/deployed, run your pentests/analysis tools" secure by design will require those skills to be integrated, too.

Little by little, chipping away at the waterfall

> the goal with DevOps was to stop doing the old "here's a build, deploy it" that caused so much wailing and gnashing of teeth

From an ops point of view that might be the main selling point. From a dev point of view a key selling point was not having to wait a month for every minor configuration change to go though change management processes.

True. It's been a while since I worked at a place where there was a traditional change management process.

I think you’re onto something. From the article…

> our role is to enable and facilitate developers in getting features into the hands of customers

The problem here is this creates the wrong kind of incentives for developers… somehow elevating the to a level where they don’t have to care about how their code works in production.

As someone that remembers being a developer back in the days of sysadmins, we were AFRAID of upsetting the operations people. If your code brought a server down, you were at least going to face some very awkward conversations. The cartoon “The Bastard Operator from Hell” immortalized that era.

Meanwhile at one company I worked at years ago - an airline - the development team was responsible for keeping the system running 24/7. Nothing makes you think more carefully about your code in production than meeting a colleague on Monday morning who got woken up at 2am by your code failing.

While I’m not arguing for hostility in the workplace, giving developers incentives to care about their code in production seems to me to be one of the things devops got wrong

Why is getting yelled at the next day more of an incentive than actually getting paged at 2am?

Well either way there are direct consequences that the author of the code will feel - which is the point here.

And usually pager duty is done in rotation, rather than you only get paged for your own code. It's one thing to ruin your own sleep, but if you ruin the sleep of the person that sits next to you, you start to think about consequences in production.

It amazes me, for example, how often I've seen developers leaving their application logging, via something like log4j, in a default configuration where eventually it WILL fill up a disk, bringing down a server, rather than investing the 10 minutes it takes to switch the configuration to say "rotation" so that only a finite amount of space will be used, or just writing a bash script to clean up old files. And this is something that's very hard to pass on as best practice without sounding condescending - like the only way to learn to take stuff like this seriously is by dealing with the consequences in production.

There's a reason people end up doing plain ops and calling it devops: it's often too costly to handle this as “a task that should be done by the developers”.

Specialization makes people much more productive, because they face the same kind of issues over and over and know how to fix them quickly. When you distribute the load in your organization, everybody is going to face problems, struggle, learn and never reuse that knowledge again.

Developers should have some grasp of ops work, and be able to deal with some ops-related issues, and take part of the design of the ops-side of the software delivery, so as to make sure the infra and deployment workflow they deal with works for them.

But yes it makes sense to me to still have people specialized in ops and infra in teams, collaborating with developers.

Basically, instead of having developers doing everything or just developing and throwing code out to an ops team, we should have developers educated in operations, working in teams with at least one operation specialist (or "DevOps engineer"). That way, you should end up with infra, deployment workflow that really works for the team and is optimized for the needs of the team.

Exactly. It's not about making developers to do more work, it's about having some "DevOps engineer" or Ops/Infra guy working close with the Dev team, by being involved on the day to day development and decision making. Instead of having to open tickets and waiting X time for it to be resolved by the Infra team (who by the way, a lot of time doesn't fully understand the applications, it's constrains, tech debts, needs, etc).

I was just reading the "Building Scalable Websites" book [1] released in 2006. At that time, "DevOps" was called SysAdmins. And there were also DBAs, Network engineers, among others.

> The way to do DevOps is to fire all of the Ops and then tell all of the Devs that they are now doing DevOps; you simply can't have it both ways,

I think this points at what happened: Startup scrappy culture started permeating new technology companies, which meant no budget for DBAs, QAs, SysAdmins and other similar roles. So decision-makers fired all those roles and ask Programmers to fill the voids. At the same time "cloud computing" started to mature, so there was a change from hardware/operating-systems tinkering to software related tinkering.

One just has to see the decline of "SlashDot" which was a very SysAdmin/Operating-System focused website, in favor of news.ycombinator and similar more software-oriented forums.

[1] https://www.oreilly.com/library/view/building-scalable-web/0...

You're right but in reality DevOps teams in 2022 are managing Kubernetes clusters and gatekeepers to all kinds of cloud services to facilitate development.

Yes. I was confused reading this article because author seems to miss an important point. Devops culture is about "you build it, you run it". Not having a dedicated devops team that tries to make developers do things. I Read a book lately on that topic : Team topologies. It explains this concept pretty well

> The way to do DevOps is to fire all of the Ops and then tell all of the Devs that they are now doing DevOps

That's slightly hyperbolic and I'd also argue that there's a fundamental error there since you throw away all your platform operations. Now you have dozens of operations engineers working in separate silos and no coordination and you've thrown away platform operations engineering.

What you really need to do is give all your developer teams pagers and point all their monitoring alerts at those pagers. If they setup an oncall rotation of the developers for their own software or they panic and hire a small team of operations engineers and hand them the pager it doesn't really matter. Then you have the problems of coordinating platform ops engineering with the ops team members in the software teams. Whoever handles Ops for the software teams acts as a kind of PM to interface with the centralized platform engineering roles which is responsible for coordinating across development teams to make things look consistent across the Enterprise.

The problem with the bad old ways was that software would write code and then toss if over the wall for operations to run, and all the shitty disk full pages and whatever other crashing software badness fell onto operations, and dev teams would each individually choose to ship software that was shitty to run and all the monitoring alerts would fall under a silo under a completely separate VP where they weren't responsible for their ops metrics and would chose to work on features to deliver for their management. Give the dev teams pagers and make them accurately feel the pain of running their software and then they can make choices about how much they want to abuse their own embedded operations people that they have to chat face to face with every morning at a standup. If you then fire centralized ops, though, you wind up with dozens of different operational fiefdoms inside of one company with everyone repeating the same mistakes and nobody doing the exact same thing anywhere.

I learned that back in 2006 before "DevOps" was a "word" and I don't know what you call it or if that is DevOps.

And it seems like Kubernetes is trying to be Conway's law applied to that. So you have DevOps embedded in Dev teams shipping containers which run on Kube clusters that provide compute as a service to the enterprise (often another company entirely) and the platform ops teams maintain those clusters. Except now you're entirely missing the communication that I outlined needed to happen between the operations virtual team composed of the platform ops and the embedded ops in every dev team. SREs will claim they don't need that any more and that old school SA operations is a dinosaur except that every now and then I see some SRE begging to know how to ship tcpdump to one of their containers to do some debugging and I know that they're dirty little fucking liars...

It was not hyperbolic, that was what happened to my team. We had two system administrators, one who specialized in Unix and one in Windows. I was the lone programmer, having escaped a previous sysadmin position. I can do it, I don't like it, but I can do it.

Then one day I was told that our two sysadmins had been traded off and we were getting two new programmers, but they wouldn't be programmers and neither would I, we were going to be doing DevOps. I had just escaped that!

The fact that someone calls that DevOps doesn't make that DevOps. That is a good example of how DevOps as a term has failed though.

Reply on news.ycombinator.com