Devs don’t want to do ops
infoworld.com
infoworld.com
Developers who disclaim responsibility, or show no interest in, how their software is scaled, securely locked down, deployed or monitored are probably not that great. Or, perhaps, they simply haven't been humbled yet by a bad incident, miscommunication, or a complete lack of understanding of how real life works.
I worked with a Microservices Architect (and developer) once, and when I asked how he intended for failures to cascade properly between services; how he would maintain cohesive state, especially if things went wrong; and how he would diagnose problems between them, he confidently told me: "That is for devops to work out."
Back in the day of desktop apps, the people who showed no interest in how their software was installed on desktop computers were more or less the same ones that are now disinterested in operations, deployment, etc.
"I just want to...."
It's terribly rude, but I stop paying attention, because inevitably the rest of the sentence can be translated to:
"... ignore my responsibilities, such as security, observability, availability, privacy, security, recoverability, documentation, scalability, and long term supportability.
I would say the context of such a sentence is usually something along the lines of:
"Because X wasn't designed with <problem> security, observability, availability, privacy, security, recoverability, documentation, scalability, and long term supportability </problem> in mind, to just do Y it is now also my responsibility to do <problem>"
Excuses are just that: excuses.
JFC, that's an incompetent.
I know a lot of people will consider that too strongly said, but I don't think so. Any jackass can succcessfully build for the happy path, it's precisely where the happy path isn't that real skill comes into play.
"Hey bob, it seems the money we sent through the system disappeared. Can we trace it?"
"Sorry no, we got rid of the reconciliation id we use across our systems."
If it's a move money issue - that's solved - transaction to credit/debit. Reject and log if failed. Maybe with a retry. Maybe from a queue.
If it's financial documents moving, they go through a serious of queues and get dead-lettered and handled appropriately.
The kicker? All of these have a customer / transaction ID and don't need some special request id "traceable throughout the system" - it's traceable as is, search for it by customer / transaction ID.
This even applies if you are using K8S for no reason. The only company I know that's justified in using K8S is an AWS/GCP/Azure competitor in the ML space needing to spin up VMs instantly at scale.
How? Just use AppService / EC2 for managed machines. Scales just as well.
"Ingress solves how to route HTTP traffic coming into your domain."
This is actually helpful and "easier" out of the box than putting nginx in front of statically hosted sites, so I will reluctantly give you this one.
"PersistentVolume solves how and where to store data."
You need a DB most of the time. Managed DBs are easier.
"LivenessCheck solves how to restart your crapware when it freezes. Just don't go crazy with it unless you actually need the fancy stuff."
Our crapware doesn't freeze. Needs a custom Health Check endpoint. Can just as easily restart the instance when that check fails.
K8S is a ton of extra overhead for no gain. We have this debate at work all the time. Great resume builder though.
It's not about scaling. It's about being able to make a docker image, run that locally on your developer laptop (even if it's a mac) and running the exact same thing in production. Sure you can use docker on top of EC2 but why would you? Just use the ready to go solution you can just buy (managed k8s) and avoid the lock-in. Again, it's critical that you use managed k8s. Don't roll it yourself on top of EC2.
> You need a DB most of the time. Managed DBs are easier.
Sure. Using k8s for your app isn't stopping you from doing that if you want. We deploy 1 database server per testing environment, which kubernetes makes easy to do. For production we use a managed database, but they're too expensive for testing environments.
> Our crapware doesn't freeze. Needs a custom Health Check endpoint. Can just as easily restart the instance when that check fails.
Mine doesn't either. But I have to run plenty of software that isn't mine.
> K8S is a ton of extra overhead for no gain
If you used managed k8s you're just trading server management for kubernetes resource management. I've migrated our system from Ansible + Cloud VMs to kubernetes several years ago, after running it for years before, and things have become significantly easier to maintain and our costs are roughly the same as before. Less vendor lock-in, too. It would take me an about an afternoon to switch between any of the major managed kubernetes offerings, and some of our clients have expressed preferences for a specific cloud.
We use .NET/React for the most part, so I haven't had an issue with Mac/PC - we have multiple teams and, for example, a team helping is all on Macs - they can run stuff fine from bash. Azure Data Studio let's them access Postgres (and would let them access SQL Server) if needed. React only has issues with file pathing sometimes, so most of the time it's fine. I find that devs also get lost in setting up Docket/K8S locally, especially when people roll scripts to set up the whole environment in one command.
Still, I am genuinely arguing to help me fill the gaps where I might be incorrect, so I really appreciate your replies.
Granted, they had entirely too many separate web services, and that was really the root of their problem, but being able to trace a request all the way through your system isn't useful until it is (meaning, when shit hits the fan).
We give them some exposure & some pick it up slowly over time. Unlike learning, say, React, most systems are pretty different. Everywhere I've gone has had very different CI/CD & different platforms for running our workload. There's a ton of fundamentals about computing & containers & building (compiling, bundling, packaging) software underneath, but usually the tools are working at higher level & those fundamentals are sparsely seen/encountered, so while a dev might start to skill up in the local ci/cd/platform, the experience only crudely translates l, they have to start way way back, if they go work somewhere else.
And then there's the question of architecting. Knowing all the varuous bits & pieces. Our apps use sql, redis, elasticsearch, an object store, an esb... good luck being able to effectively understand & operate all these machines, newly minted senior sev.
Coding is way more insular.
Without those generic abstractions, you need a ton of ops people. Enough that there are at least a couple members of ops with in-depth knowledge about each app, and each ops member can only realistically know <5 apps that well. Really, you need several ops members to know each app that well, because that style of ops tends to burn people out rather quickly and they quit. Ops also has to be intimately involved in the design process, and have some kind of veto power to say "you can't build a stateful app, then throw it over the fence and demand we somehow make it scalable".
With the generic abstractions, you need fewer ops people, but they really only know the generic abstractions and can't help diagnose errors via log messages.
A lot of (bad) devs want a world where ops is just held captive by dev. Ops has no authority to say no to the design process, and no ability to say that some things aren't possible with the application architecture. It just doesn't work. Even if ops can somehow do things like remove state from stateful applications, everyone capable of doing that work is going to quit. It's not fun or interesting work to do, it's exhausting and frustrating to expend man-months of effort on a problem that could have been solved with man-hours or weeks of development effort.
I don't want to do ops, but I will if they're holding back development.
> “Fundamentally, it’s what happens when you ask a software engineer to design an operations function,” Ben Treynor, vice president of engineering at Google and the godfather of SRE, is often quoted as saying.
A foolproof plan. Much like hardware, software is generally known for its reliability, interface standardization, and design stability.