You build it, you run it [pdf]
equalexperts.com
equalexperts.com
As a dev, I do know a bit about infra topics (just because I’m curious) but not enough to be oncall on Sunday mornings.
- clear error messages (with the actual data that caused it, minus privacy-sensitive information) - possibility of reloading configuration without having to restart the whole application (configure-once-at-construction-time is not enough; not every application is a web application with a php-like lifecycle)
- atomic check pointing and restoring of long running operations, without leaving a trail of partially truncatd temporary files
- security: you do not know how to write secure software until someone hacks your software in an unexpected way. This includes knowing how actual infrastructure configuration affects the running software (symlinks, permissions, capabilities)
- performance: unless "scale" a software means blindly increasing the number of replicas, there are so many architectural lessons to be learned studying flame graphs of actually running software that it is hard to imagine to acquire those expertise without making your hands dirty. At that point, it is even possible that your software works so well it no longer needs so much hardware
If your application is a client-side GUI you may not want to restart it on reconfiguration. This applies to other things as well: think about having to restart your reverse proxy/load balancer each time you add a new route, or rebooting your OS each time you reconfigure a systemd unit, or reinitializing your embedded HW because something changes. The possible incarnations of SW are very much varied, and not everything can be solved in the same way.
Some systems are simple enough that a single person can understand everything about the entire stack. Hell, I still run downright ancient LAMP stacks on colocated servers where I'm the only one familiar with the code base and I interact directly with the dude on the floor when I need a hard powercycle.
But there are a lot of systems in which the ops side is incredibly complex (think compute-intensive features with constant churn in the underlying hardware, or fleets of embedded devices that are always failing in new and fun ways, or finicky lab equipment, etc. Have you ever run a fleet of machines each hooked up to and controlling piece of $x,xxx,xxx equipment, where each of those pieces of equipment is a snowflake? And where all the data from that fleet of equipment gets pulled off the machines attached to the equipment and sent over to a computer cluster the size of a small DC?).
There are also lots of systems in which the software requires a lot of specialization, either technical or subject matter (in the above case: the software running on that cluster was written by and requires the expertise of phds in a few different natural sciences + CS, and the hardware at the time was so bleeding edge that we were essentially the chip maker's and system integrator's beta testers.)
The real issue is when these two types of systems very often overlap.
And, like, this sounds special, but it's really not. There are projects like this is every sector -- life sciences, finance, tech, basically anything in a hospital is a 50/50 chance of being like this (mostly due to externally imposed complexity), agri of course, even property insurance. I'd say that "ops and software need separate on-calls" probably describes most successful software products/orgs.
Even if one person did have enough time in their career to learn how to be the oncall for both the software and the ops, there's no way they could keep up with all the moving parts of both jobs to be able to do both jobs at the same time. At least, not without burning out pretty quickly. At some point these have to be separate jobs because just keeping up with all the moving parts well enough to do an oncall shift is by itself a >60 hour/week job. And that's just to stand still in the feature set. At some point we wanna get new work done.
FWIW, I haven't been paid to work on the former type of system -- where both aspects are simple enough for one human to first grok and then more importantly keep up with -- in a long time. I sort of
From personal experience, either the company is trying to be cheap and not hiring an SRE/devops until it is too late or misunderstands what they bring to the table. You need multiple disciplines working together to offer good, reliable and secure deployment of a complex app.
This is true for a lot of disciplines: nobody understands everything in cryptography or graphics or networking. But most people have enough understanding to model how the other pieces connect. You have to treat a lot of pieces as black boxes.
I think what is the important point I agree with the presentation is that things do not end with feature design and implementation is finished. Part of the feature is the feature to be made available. Feature availability is part of the product.
But there is considerable cost with "you build you run it." That means you are chained to it and you can not move ahead: if you keep responding to incidents how are you different from the case of a CEO that does not delegate and tries to code, raise money and do customer support and administrative tasks. You find yourself never moving forward. Delegation and collaboration are important.
This way each individual App Team doesn't have to become a PhD at Kubernetes and the Platform Team (aka "Ops") doesn't have to become a PhD at 50 different apps.
Of course… the “platform” can and probably will become crap over the long run. Or it’s too doctrinaire and opinionated inflexible and new people don’t like it. Then you end up with alternative platforms popping up over the years, and now there’s multiple ops teams working in different silos (legacy and new etc) and eternal migrations. The circle of life!
I've seen this far too many times.
I quit, your startup fails.
A bunch of sorry chump ass latecomer teams who are still run by useless ignorant product chumps who cant understand shit & just want features features features.
Oh sorry, everything is shit & everyone who built this shitpile split, but you cant really grok any of that can you? Making the engineers the sole & pertpetually responsible ones is a shit show. Common and trash & so typically ignorant.