Hacker News Reader

Comment by itsmemattchung | Hacker News Reader

I was one of the biggest proponents of DevOps (even started and the DevOps orange county meetup) but ... after working at AWS for 5+ years, my entire perspective shifted.

While DevOps does work in some organizations, sometimes what you really need to do is operationalize your software development team. Don't get me wrong, system developers/engineers/operations are needed but for sometimes for a small four person start up, the best investment is improving the development team's operational posture (e.g. run books, alarming, monitoring).

View on HN

> for a small four person start up, the best investment is improving the development team's operational posture (e.g. run books, alarming, monitoring).

Oops, I thought that was DevOps.

That's actual-devops. Grandparent poster has worked at firms which have abandoned all its lessons and just use it to mean something like "regular old ops teams, but they use software". So he's using the term like all the big companies do, basically.

This is also where you will find things like a "CI/CD" team that doesn't do continuous delivery: they just build Jenkins pipelines, and run them as part of the manual cluster bringup every 2-4 weeks. Or "RESTful" to mean "APIs which use JSON and HTTP verbs" regardless of any hypermedia. And let's not even get started on "agile"...

'word', as they say

Here's a 400 page book and a 12 week training course on all the processes we used to implement Agile's "People over process" principle.

Me too.

In my experience it went poorly because us devs are now handling our own deployments and building alerting systems and such, but the "devops" people won't give us the level of access necessary to do that with tools designed for the job, so we end up building custom ops-related tooling into the app and writing backdoors that let us use it without having wake the ops folk.

When you're accustomed to having to build everything you need, and you're tasked with ops work, you end up reinventing a lot of wheels in a rather weird way and compromising your app in the process. Or at least that's what we did.

This really substantiates the OP’s claim.

Maybe we need to come up with a new term to be more specific. “DevOps for developers” vs. “DevOps for operators” or something like that. (DevDevOps va OpsDevOps?)

I always took DevOps to be some combination of 1) applying dev best practices to ops, which could still in theory be done by a separate team, but more 2) unifying dev and ops in one team/role, perhaps enabling developers to own it by providing tools and abstractions that developers can work with, and workflows that unify the processes and improve collaboration by lowering cycle time.

It seems that the “bad” DevOps is OpsDevOps where the old ops team just gets some rebranding, and maybe some new tools like Terraform. I’ve always found the “DevDevOps” approach to be quite empowering. Giving developers a tool like k8s so they can abstract away the hard half of ops, and understand the rest.

Yeah, to me, the core of the ‘DevOps’ philosophy is that the dev teams owns the product end to end, meaning they write it and are responsible for it in production.

There seems to be an increasing disconnect between the marketing value of a term ("DevOps") and what we're actually supposed to be doing, which might best be described as blameless accountability.

Even in information security it basically revolves around the same things before you can even do any sensible work and improvements to processes.

  - Know what you have
  - Know where it is
  - Know what it is doing
  - Know what it is supposed to be doing

If you don't have any of those pieces you're just guessing and hoping for the best. For software that might be "know what services you are building, know where you're running them, know how they are doing, know how they are supposed to be doing". But if we call that "DevOps" is suddenly marketable.

The issue of course is those things are hard.

They are only hard if they don't get the same priority and attention you'd see in regulated situations. Replace software or information security with "bars of gold" and suddenly it's not so hard.

That's because marketing why you should know the what/where/how/why of your gold to business decision makers is apparently much more palatable than abstract concepts found in IT, even if the value is the same.

If you get the resources (including time, money and manpower) do just do it, it can be done. But you usually don't get all three (and maybe not even one), and therefore you're stuck with a task that cannot be executed well.

Did general aviation become cheaper or more available to folks when it got more heavily regulated?

Fatality rates dropped, but it basically froze the technology and capability into the late 70’s/early 80’s and locked in just a few companies. The overall number of people working in that industry is tiny now. It’s almost a complete dead end.

It’s the classic trade off - cheaper, better, faster. Pick any 2 if you’re lucky. Most only get to pick 1.

Yep, and that is generally also why IT is the way it currently is. Most projects aren't all that complex (in business logic terms), it's mostly CRUD interfaces and a fancy viewer for the end users anyway. But industries that are still on mainframes are almost completely the opposite where every single element is tagged, catalogued, tested, described and traceable at any point in time. With a price, rigidity and lead time to match, of course.

On one hand we don't all want to buy mainframes, on the other hand, doing it as cheap and unmeasured as possible makes everything flakey, demanding on individuals to keep it running, insecure, and opaque to everyone involved. Too bad there isn't really any middle ground, it's almost like it's either "pretend to be a mainframe" or "do it as cheap as possible".

Regarding the aviation analogy: it definitely doesn't become cheaper due to regulation (it did become cheaper due to subsidies), but I'd rather have a system that's regulated (or forced) into safety than having it be optional and left to a sociopath (i.e. a for-profit publicly traded company - all the benefits of pretending to be a person but none of the responsibilities or limits to govern it - except the law and regulation).

Isn't that list just "Ops" ?

No, "Ops" is legacy siloing of departments and by extension, tasks. Imagine driving a car, but it takes one person to do the steering and another person to do throttling. Seems pretty dumb, doesn't it?

On larger vehicles like ships, you have exactly that "legacy siloing" of responsibilities. You don't need the person responsible for navigation and steering run down into the engine room or to the rudder to check whether everything works as expected there.

True, but the analogy for a ship would be: the person cleaning the deck isn't allowed to get his own bucket, that's the bucket operator's job. Pretty silly again.

Late reply, but, to me the difference between Ops and DevOps is whether or not you need to contact IT to get your original list of:

  - Know what you have

  - Know where it is

  - Know what it is doing

  - Know what it is supposed to be doing

> the best investment is improving the development team's operational posture (e.g. run books, alarming, monitoring).

this is devops

The original definition of DevOps was what you describe in your second paragraph - development teams being responsible for operations.

Now DevOps means "We've created a DevOps team! They do DevOps!" Frequently its the managers of old school operation teams changing their names to DevOps teams to get a promotion.

It may seem pedantic, but it's an important difference: that's _not_ what the original definition was. Instead, it was a cross-functional team of developers and operators being responsible for a whole system.

Yes, I should not have said a "development team", as it makes a distinction between "people who create new software" and "people who run that software in production environment". Your definition is better.

I always liked this early definition of DevOps:

"Giving developers operational responsibilities has greatly enhanced the quality of the services, both from a customer and a technology point of view. The traditional model is that you take your software to the wall that separates development and operations, and throw it over and then forget about it. Not at Amazon. You build it, you run it." - Werner Vogels, Amazon CTO (2006)

DevOps meant that Developers are also responsible for Operations. But that meaning is lost to time, just like hacking meaning programming.

> development team's operational posture (e.g. run books, alarming, monitoring.

Isn't DevOps in large part forcing the developers to have responsibility over the code they push and giving short and frequent pushes to easily rectify problems?

Making developers do support work reduces support in the long run because they'll get frustrated and automate solutions.

Yeah, I feel like the term has been taken over to mean using kubernetes and having continuous deployment pipelines for your microservices.

My understanding of root of the term is about ownership and responsibility, where the team that develops the app also owns how the app deploys and runs in production, including monitoring and responding to issues.

If the team decides that the right way to deploy their app is once every two weeks using a shell script and a zip file, cool. As long as it's documented and monitored so you know if it's working.

Yeah, what's worked well for me is having ops or security people draw up guidelines (for security, it may be something as trivial as require ssh key auth, for ops it may be: write a script that matches an existing deployment procedure and we can get your app up and running very quickly).

And/or become burnt out and develop a severe ‘learned helpness’ complex. It depends on the environment they’re in, and if they have the tools and support to be able to fix issues.

Many companies, they don’t.

Which definition of DevOps are you against? and what changed your mind against it?

> for a small four person start up, the best investment is improving the development team's operational posture (e.g. run books, alarming, monitoring).

But that's literally what DevOps is? DevOps isn't a special DevOps team that writes terraform.

Reply on news.ycombinator.com