Comment by smackeyacky | Hacker News Reader

Comment by smackeyacky | Hacker News Reader

Parent Full thread

I'll give you a concrete example of how it goes so very wrong.

I worked for a bank for a short period of time, where the development team had gone a bit far with a shotgun approach to moving their systems to Azure. It was pretty hard to find things and as their approach evolved, the older conversions weren't revisited. Most of that team quit. New team is brought in (including me) which had some excellent engineers with years of experience in system architecture, how to make Azure work better. So we tried to homogenise things around a more reasonable approach.

This caused operational problems to worsen (understandably) but it was a short term pain, long term gain thing we couldn't be allowed time to do. So the CIO decided to take all the sysadmin/devops type work and give it exclusively to the system administrators. Who weren't developers. They fixated on one particularly narrow solution for deployment. To make it easier for themselves, but one that didn't really address the bigger picture of how to make it easy to monitor and deploy etc.

Anyway it ended up a disaster. The development team in their newly narrowed roles struggled to make their systems fit in the rigidly defined holes. Operationally it was no better and sometimes worse, but there was absolutely no compromise on how things should work or any consultation with the devs at all, ever.

I no longer work there. If you're going to do devops you have to listen to your experienced engineers, not the snotty kids who think clickops is engineering.

This is a valid story, and I have no doubt it’s real. I’ve heard and seen many stories like this that have happened.

But… I’m going to say the dirty, quiet, and unlikable thing out loud.

That had nothing to do with DevOps or its philosophies, processes, or patterns. That was bad leadership from the top down plain and simple. It’s likely not even the individual engineers faults. It’s leaderships fault for not setting clear objectives, implementing them, ensuring that the engineers had a real plan before beginning, and making sure no individual was too in charge of things.

Leadership in your case was likely career management who knew very little about technical items. Managers who were technical were probably shot down for not playing politics properly, not producing the correct “metrics” and “kpis”. So they moved on.

That’s a company culture issue that has little to do with tech.

I think the comment you are replying to is pointing out the terrible ramifications of just taking a random sysadmin team and expecting them to work in a full blown devops role. I too have seen similar situations play out, and in slight reverse - in their scenario the devops guys were calling all of the shots, ive also seen (and have lived) the opposite where devs call any and every shot on devops stuff and the devops team is treated as little more than glorified IT tech support but for devs (understandable how a business could come to a conclusion about why that makes sense, given how they perceive the sys admin -> devops progression). This sounds like it’s what you may want but it has its own sorts of problems - devs dont always know the right thing or what to ask for or what not to ask for. Really good devops guys are experts at guiding these conversations collaboratively.

I have known and worked with some really great former sys admins gone devops. I am working on mentoring one right now, but I have to be kind of insulting about it and be like “forget everything you knew before it probably won’t help now” which sucks because sys admins do form pretty decent understanding of OS’s, databases, networking, etc. however, when it comes to the code part and more importantly taking all of these concepts and applying them to reasoning about infrastructure code and complex systems is very hard for most people and you have to take a “im a total newb” mentality a lot of people dont seem easily capable of doing.

Valid assessment. The CIO in question did not have a technical background as was far more interested in fostering cliques than good engineering practices. The culture of the place was toxic as a result, it rewarded surveillance and not performance. It promoted fantasy land project managers who ignored reality.

Still, it made me very wary of the idea that devops is separate to development.

Managing relationships with project management and the difficulty that entails with most devops orgs where ive been is a symptom of the same problem I feel is being circled around. Good devops managers manage around these realities really well, can think of some specific examples in my head - it’s very much a several way conversation that should punish silos. however in practice it seems to reward it. If devops worked with and closely with devs you do not need layers of semi competent project managers in between acting like glorified note takers.

I worked in finance on the other side of the pond - developers wanted to constantly bring in and use new services but also didn't want any of the responsibility or the work needed to make compliance happy (or even in that particular company shoulder the costs). When me and other folks where brought in it to fix the "cloud strategy" it was a complete shitshow and heads actually rolled when we wrote a tool to assign costs to applications. But we had to start almost from scratch and limit usable services as we developed strategies and blueprints for each...

The complete, unapologetic desire of devs and security teams (but also many infra teams) to not have any kind of ownership was horrifying to me.

In the end there's not a single solution or strategy, it really goes back to the organization and where your weaknesses and strength are as an org. If you have a gazillion consultants following the "best practice" of the day and exceptions on top of exceptions you are dead, devops or otherwise. You will still make billions if you are the right company though regardless of your software practices, so...

Reply on news.ycombinator.com