I always do a better job of writing logic when I sympathize with the victims. Some of those late night calls are devs setting ops up for failure. It was meant to close an open loop.

> The fun part is Ops wants us engineers to help more, but they don't even trust (and some don't know how to run) my Terraform or CloudFormation scripts (I let them choose which), so they just read the tf/yaml and click AWS buttons manually. Often making lots of mistakes.

Yeahnah, you're part of the problem.

If you've not documented it properly, and more importantly, if the team are weary of prodfucking TF/CF scripts, there is a culture problem. You both need to understand how that works, if they don't have enough time, then you're understaffed, that needs to be fixed.

I worked at place that was going through a really painful ops->devops->dev oncall transition. Ops still had a shit ton of power, but nobody included them, and just shouted at them when stuff went red. It took a huge culture shift, and devs sitting in ops and doing a shift or two, to sort it out.

Because that is the software developer/systems engineer division that already existed 20 years ago.

The idea behind devops was to be a single team.

Because it's just a hip way of saying you have a sysadmin team. How can something or someone be "devops" if all the developers are in one team doing dev tasks, and the ops in a separate team doing ops stuff?

The thing i have come to dislike is that in both companies i work for it lead to strange working relationships between ops/eng. anything that breaks in the deploy pipeline is now “the devops team problem” and they normally act like it’s also your fault. In the past this has meant people messaging me saying they are blocked because their dockerfile fails in CI. Then you walk through it with them and it’s also broken locally and it comes down to them pinning to something like latest tag and the image “shocker” changed. But now you are fixing it and after you go remind everyone that you should pin your image to as strict as possible image tag. But no one cares because they can move fast and break things and when it does blame it on CI and now it’s not their problem.

Anyway this is just one tiny example but the same underlying thing starts happening in all parts of the development cycle. “My deploy doesn’t work” … well your program has an error and crashes on startup but looking at the logs would be too much work. Let’s just say we are blocked and log off.

As you can see this is a culture issue but it seems to be common when you have a devops team. And even more common when management (read senior management and up) think that developers “should just be developing”.

Anyway thats my rant.

That’s fine, it just has nothing to do with DevOps then. The whole point is that these are not separate teams.

So in other words, you made silos. More throwing things over the wall and having it be someone else's problem. That's fundamentally the antithesis of DevOps.

People do not fix the pain they don't see, and managers doubly so. By shifting operations off to a "devops" team, you just end up with developer teams that don't know about, and don't care to fix, the things that make it painful to deploy or run their code in production. Those that have to deal with the pain rarely have any political leverage they need to even enact change. There's zero motivation for either the developers writing the software _or their management_ to fix it. Dev manager incentives are purely aligned with getting code and features built.

To give a nice real world example, building out new regions for AWS services used to be a pain in the arse (since I left I hear some major top down initiatives have fixed that). The service team I worked for, almost doubly so painful for region build. Standing up an entire region was something like ~50 days of engineering time. Not because it was especially hard, per se, but because the ops team always had to wade through piles of refactored shit that introduced new circular dependencies, or wasn't documented, or broke other aspects of any automation we could produce. (I remember one DynamoDB schema overhaul, where we did a complicated migration. Next region build, service components wouldn't stand up because we'd used the automation to create the new table schemas that had replaced the old table schema entries, but the service code still expected to see the old schemas in the right format. It took several months to get the dev manager to agree to engineers spending time cleaning up old schema references from their code base.)

The dev manager incentives were not aligned with making it easy. It wasn't their engineering resources being consumed, and they had features they needed to ship. The developers never changed the way they worked because even though they were told the issues (and even though they were, virtually to a person, great people to work with, considerate etc.), they never experienced the pain so they never thought about those consequences of what they were doing when they were doing it.

Ops manager never had any leverage, because every single time we managed to get a region launched in time as to not be an issue, and every time things were down to the wire it was clearly as a result of the ops team not building the right automation :eyeroll:, rather than that the dev teams were making some kind of brand new nightmare each time.

It wasn't until the ops team had lost enough members of staff to that BS, and region build shifted on to the developers themselves, that suddenly most of the problems went away, never to appear again. They were aware and conscious of the pain certain decisions would cause because they had felt it themselves, and they made sure not to introduce more pain.

DevOps is about making sure incentives and politics align to make things overall better. Shoving it in to a silo is literally the complete opposite.

It's like saying you have a self-service waiter.