Hacker News Reader

Comment by 0xbadcafebee | Hacker News Reader

They're doing that thing that happened to DevOps. It started out as a guy who wanted a way for devs and sysadmins to talk about deploys together, so they didn't get dead-cat syndrome. It ended up as an entire branch of business management theory, consultants, and a whole lot of ignorant people who think it just means "a dev who does sysadmin tasks".

Abuse of a single word to mean too many things makes it meaningless. SRE now has that distinction. You've got SREs who (mostly) write software, SREs who (mostly) stare at graphs and deal with random system failures, and now SREs who use a framework to develop a complex risk model of multiple systems (which is more quality control than engineering).

View on HN

> a way for devs and sysadmins to talk about deploys together

My take on this as someone who has 14 years of dev and 7 years of ops experience: DevOps is a flawed concept.

The problem never was the lack of communication between the devs and sysadmins, it’s just a symptom.

The root cause is that the management puts pressure on the devs to innovate and deliver as fast as possible, and puts pressure on the ops to ensure that the system is stable, reliable, scalable, it has a 99.95% uptime and any issues will be solved by the on-call.

So these two groups have conflicting interests and when this leads to conflicts and arguments the conclusion is that they just don’t want to collaborate/communicate.

There are many departments at a company that have conflicting interests and can interfere with each other. If DevOps was a real thing there would be a need for LegalOps, DevSales, DevProduct, HROps, etc..

Paralegals, legal clerks; sales engineers; product management, project management, TPMs; HR coordinator, HR administrative assistant...

I'm not sure your examples say we don't need DevOps!

> Paralegals, legal clerks; sales engineers; product management, project management, TPMs; HR coordinator, HR administrative assistant

The point wasn't that there are multiple job titles in different fields.

> I'm not sure your examples say we don't need DevOps!

The main assumption of DevOps is that the friction between developers and operations is the main cause of slow delivery.

Well this friction can be happen between dev and legal, and dev and sales, and dev and product, and ops and product, and ops and sales, and ops and finance, etc.

The companies that need DevOps have a much deeply rooted problem: the lack of collaboration between any department (not just dev and ops). Therefore DevOps won’t solve their problems. And companies that don’t have this problem won’t need DevOps, because they don’t have problems with collaboration.

> DevOps is a flawed concept... conflicting interests... the conclusion is that they just don't want to collaborate/communicate.

Wrong! This tendency is exactly what DevOps explains is the natural state of affairs without DevOps principles. The solution that DevOps advocates for is that such conflicting interests must not be expressed in meetings (where the culture conflict ensures they will get nowhere) but rather expressed in code. Infrastructure must be in code, deploys must be in code, testing must be in code, builds must be in code, policy must be in code, and the implicit pipeline with all the handoffs between teams connecting them all must also be in code. This makes everything fast (at least in comparison to manual processes), and makes everything explicit (in code) so that people can reach outside of their natural organizational silos to propose changes elsewhere in the pipeline, i.e. infra-focused engineers can add failing tests to prove the existence of a bug, developers can add infra that they need, QA can increase infra resources to ensure that sufficient resources are available for expected scale.

The problem most organizations have is that they're not actually willing to force everyone's concerns to be written in code, and people are forbidden from reaching outside their silos. Usually this is due to poor hiring and training practice, e.g. "QA doesn't know how to write code" or "we can't let developers touch security policy". Sometimes it is due to leadership itself misunderstanding DevOps ("developers are forbidden from touching production").

> If DevOps was a real thing there would be a need for LegalOps, DevSales, DevProduct, HROps, etc.

There is such a need, and it is generally being fulfilled by the systems in place. Small example - HROps would be ensuring that changes in the organization (i.e. people moving teams) accurately results in proper loss (of old) and gain (in new) privileges. This is done with integration between HR systems (the system of record as to who reports to whom and what their responsibilities are) and Active Directory or Google Workspace / Google Groups ensuring that people are automatically moved between the groups to which permissions are granted.

> Wrong! This tendency is exactly what DevOps explains is the natural state of affairs without DevOps principles. The solution that DevOps advocates for is that such conflicting interests must not be expressed in meetings (where the culture conflict ensures they will get nowhere) but rather expressed in code.

Yes, there is the theory, principles, advocates, etc, etc. And there is the reality, and based on many-many years of experience as a dev, as an ops, and as a manager at from small-sized to enterprise companies, the reality isn't even close to this.

> Infrastructure must be in code, deploys must be in code, testing must be in code, builds must be in code, policy must be in code, and the implicit pipeline with all the handoffs between teams connecting them all must also be in code.

This is not DevOps, and you don't need DevOps for this. This is just about having an engineering mindset.

> so that people can reach outside of their natural organizational silos to propose changes elsewhere in the pipeline

Do you know how many times have I seen a developer touching Terraform code, ansible playbooks, or pipelines described as code? I am not saying that it never happened, but it was a rare occasion.

> The problem most organizations have is that they're not actually willing to force everyone's concerns to be written in code

I managed such an enforcement and change. It did not solve the cultural and collaboration issues.

> HROps would be ensuring that changes in the organization (i.e. people moving teams) accurately results in proper loss

This is just a matter of SoPs and workflows. It has nothing to do with the topic.

No offense, but your comment is a perfect reflection of why DevOps is a flawed concept. You are talking about enforcement of everything described as code, advocates, principles, etc.

If there is a good culture and collaboration the infrastructure as code, the advocates, etc will come naturally. People will find a way to collaborate. But not the other way around, these won't fix the culture.

> If there is a good culture and collaboration the infrastructure as code, the advocates, etc will come naturally. People will find a way to collaborate. But not the other way around, these won't fix the culture.

Sure I'll agree that DevOps is a culture change. You're right that exposing everything as code does not, in and of itself, get people to start having discussions around code or proposing changes to the code. If people are used to the only way to something getting done is by waiting for meetings and opening tickets, then the fact that every team has code in a pipeline doesn't inherently change that. But without ensuring that the code is there first, you cannot enable the cultural change; it is the fact that the code is available is what makes possible an expectation from management to first go look at the code and try to understand it and have discussions around the code and even propose changes to code outside of your domain.

Without the code exposed, you can have very "collaborative" cultures. They look like people whose calendars are booked straight through for weeks, where they spend the whole meeting trying to get their heads aligned, sometimes going over basic details, where at the end, hopefully, you get a ticket opened for them or their team. You spend so much time collaborating that nobody actually has any time to get anything done.

While reaching across silos seems good, in theory, my experience is there's just too much breadth and domain knowledge for it to work consistently.

Sure I know application code and have worked with a handful of frameworks, but if I'm enforcing infrastructure or performance concerns and implementing across a handful of different services, it's extremely time consuming getting up to speed in each repo and understanding the subtleties and patterns between each one.

I can optimize queries and debug performance issues but the usual roadblock is understanding what the code is _supposed_ to do and whether an optimization provides the correct results (which is not always clear from tests, assuming good ones exist)

The argument isn't so much that you can reach across and do everything independently, it's that seeing the underlying code makes collaboration easier. Sometimes it means you copy/paste links to the code into Slack to ask about them. Sometimes it means you open an MR and ask for review because you're not sure. But both of those are highly preferable to opening a ticket for a different team and waiting around for maybe someday they'll get around to tackling something that a) the Business didn't ask of them and b) they didn't open for themselves

> You've got SREs who (mostly) write software, SREs who (mostly) stare at graphs and deal with random system failures, and now SREs who use a framework to develop a complex risk model of multiple systems (which is more quality control than engineering).

This was always the case or at least going back 15 years or more highlighted by the so called “treynor curve”

NB: The Treynor Curve is named after Ben Treynor and his ideas. Ben Treynor's name changed to Ben Sloss a few years back, and Ben Sloss is one of the authors of this article.

Without resorting to any "big-D Devops" definition, I have almost always seen devops referring to "supporting / running the code you write", and have never encountered the definition where dev and ops were 2 different roles. That was what things were like before devops, and coordination on product support and planning wasn't great, hence devops.

A lot of organizations simply renamed the functional area of "systems administration" or "systems engineering" to "DevOps," and at many of these places, "DevOps" is the new name for the group that software developers will throw stuff over the fence to.

The issue with the above names is that they can be applied to a domain or area of practices, or an organizational boundary. In a non-trivial number of organizations, "DevOps" is viewed as a support entity for one or more software development teams, versus software development teams practicing "devops."

This applies to many of the *_Ops names in fashion during the past five years or so.

After almost 10 years in the systems engineering / administration / devops / cloud etc space all I can say is:

The biggest improvement that devops brought is that it made managers feel dumb, outdated and scared because they were not "doing devops" while everybody else was, so they kinda started listening to sysadmins and what they had to tell.

Uh, devops engineers did not come out of nowhere. They did not come out of the ground like mushrooms. Most if not all the "devops engineers" i know are just former sysadmins. They were already willing to do whatever devops was supposed to be, it's just they they were largely ignored.

Writing this I just realized that maybe the best way to obtain organizational change is to make management and upper management feel stupid and outdated. Interesting.

Hence they have all hired “head of AI” in last 18 months.

"What is your AI strategy?" -> "We're hiring a Head of AI"

It’s the 2020s version of the old IBM line… No one ever got fired for hiring a head of AI.

I'm Ops type person so I work at companies where there is a split between the two. Ops is a skill not all developers have or frankly, not even mindset to properly do so you will need a team/person to do it. Generally companies don't like the cost of embedding Ops person into every team and that can create redundant work so they form a DevOps/SRE team.

Good resource for different types of teams is here: https://web.devopstopologies.com/

The reason you never encountered the second definition is two-fold:

1) There is no formal academic education behind the concept (that I'm aware of). If you do a CS major, nobody's going to explain to you the accumulated 15 years of practice and knowledge around the concept.

2) Due to 1), people just repeat what other people tell them. It's like a long game of telephone. It turns out most software development today is just a game of telephone between devs (and now AI). So almost everyone is misinformed.

The Wikipedia page for DevOps is the best generic starting point if you want to know more.

If you want to know more after that, there are a number of books and blog posts. Jez Humble, John Willis, Gene Kim, Patrick Debois, etc are the people to read. It's a much larger body of knowledge than you might think. Almost none of it has to do with devs supporting/running what they write (that's a small subset of a larger category, and there's multiple categories of 'stuff')

I think failure mode analysis is definitely part of engineering

Never heard of dead-cat syndrome. In case anyone else wonders:

> There is one thing that is absolutely certain about throwing a dead cat on the dining room table – and I don't mean that people will be outraged, alarmed, disgusted. That is true, but irrelevant. The key point, says my Australian friend, is that everyone will shout, "Jeez, mate, there’s a dead cat on the table!" In other words, they will be talking about the dead cat – the thing you want them to talk about – and they will not be talking about the issue that has been causing you so much grief

https://en.wikipedia.org/wiki/Dead_cat_strategy

I actually don't think that's the dead-cat-saying that the parent is referencing. I think that it's this concept http://itskeptic.org/dead-cat-syndrome.html

I am also unfamiliar though and I'm reading up on it right now.

That's the one. Only old fogies (like me) know it I guess. It was the thing we all referred to as the impetus behind DevOps, when it became a thing a decade ago.

i’m not an old fogie and i kind of guessed “throwing a dead cat over the wall” was what you meant.

although i was actually taught DevOps proper before it got super fashionable.

maybe that makes me an old fogie nowdays? :/

> devs and sysadmins to talk about deploys together

Not quite, it was the merger of development and operations responsibilities.

I.e. devs who would sysadmin / sysadmins who would development.

Reply on news.ycombinator.com