Comment by dasil003 | Hacker News Reader

Comment by dasil003 | Hacker News Reader

Parent Full thread

At a certain scale mature software companies do in fact have dedicated incident managers, and dedicated SREs who work primarily on stability from a more systematic perspective. However they still need support from the application developers due to the nature of software.

In the old days operations tended to be very isolated in much the way you are proposing. The problem with this is that stability depends very much on the software, so over time operations folks would be extremely defensive and impose all kinds of constraints on what software could do, and the software engineers would be frustrated that they couldn't do things efficiently. Imagine how firefighters would feel if construction workers had a tendency to randomly leave explosives and gas cans hidden throughout new construction and then waltzed off to the next job while the firefighters had to deal with the consequences.

At the end of the day, devs need to have some skin in the game or it's a recipe for disaster.

The "skin in the game" argument is, IMO, not compelling. It is clearly possible to have stable software services delivered by separate Dev and Ops teams that communicate using a will defined software interface -- look at any app that uses Heroku or a similar PaaS.

But, as we know, useful software interfaces are difficult to define well and, once they exist, they tend to be the most inflexible part of a fast-changing system. It is always better (though of course more expensive) to control both sides of an interface for this reason.

The "skin in the game" argument elides this fundamental reason and substitutes one that implies all of this is the fault of lazy devs, which isn't (generally) true IME.

This argument doesn't hold water. Both Heroku and teams that run apps on Heroku have their own on-call teams. Yes, you can build stable interfaces and separation of responsibilities between infra and business services, but someone still has to be responsible for the business services stability.

Right but I meant my point to be that the people on call don't have to be literally the same people developing the application code. In fact, they never have to communicate at all, because they can coordinate at arms-length with a well-defined software interface.

Edit: I missed the part where you say the Heroku customer has their own on-call team. IME this is not true. The whole reason to use a PaaS like this is to avoid having an Ops team. Sometimes orgs outgrow their PaaS and keep using it anyway, but this isn't necessary, only a historical artifact. These orgs would likely save more money and get a better result by going with normal IaaS or racking their own servers and, in fact, are probably actively switching to that model.

The juxtaposition of "this is how it's done because otherwise those that fight fires impose restrictions that those that build don't like" and "construction workers Vs firefighters" seems to be undermining your point...

In mature industries, there absolutely are plenty of regulations in place to make sure that builders don't make responders' life harder. That doesn't mean that the responders aren't needed, but the fact that the software industry as a whole decided to go all "response is the only thing we need for most things" is evidence that it is not mature.

Sure, any point can be undermined by overgeneralization and bad analogies.

The nature of software and physical construction is different.

Reply on news.ycombinator.com