A Eulogy for DevOps
matduggan.com
matduggan.com
It really cuts to the heart of it when you looking at the “devops cycle” diagram with “build, test, deploy” …and yeah, those other ones…
I remember being in a meeting where our engineering lead was explaining our “devops transformation strategy”.
From memory that diagram turned up in the slides with a circle on “deploy”; the operational goal was “deploy multiple times a day”.
It was about speed at any cost, not about engineering excellence.
Fired the ops team. Restructured QA. You build it you run it”. Every team has an on call roster now. Sec dev ml ops; you’re an expert at everything right?
The funny thing is you can take a mostly working stable system and make fast thoughtless chaotic changes to it for short term gains; so it superficially looks like it’s effective for a while.
…but, surrrrpriiiisssseeee a few months later and suddenly you can’t make any changes without breaking things, no one knows what’s going on.
I’m left with such mixed feelings; at the end of the day the tooling we got out of devops was really valuable.
…but it was certainly a frustrating and expensive way to get there.
We have new developers now who don’t know what devops is, but they know what containers are and expect to be able to deploy to production any time.
I guess that’s a good way for devops to quietly wind up and go away.
When I do a commit prettier is run. When I PR into develop other dev’s review the code. When its approved and completed a build is run (including all tests and sent to Veracode and SonarQube for analysis). The build is deployed to the development container where the developers can smoke test and then when SQA is ready promote to Integration for real testing. Fail fast! If the tests fail, we can fix them. If code quality goes down, we can fail the gate === no deployment. Without DevOps and CI/CD these steps get skipped.
Not sure where the idea that software is released to production multiple times per ever came from. Yes, the develop branch should in theory be ready to be promoted at any time but we know that is not reality.
Any system where you maintain a separate development branch is one where you're invariably going to be asked to cherry pick feature B to production but not feature A. This is a problem because you tested everything on a version of the code where B follows A, but now B needs to stand on its own. Can it? Maybe. But you didn't test that.
With trunk-based development you have one main branch that everything goes to all the time. If you need something to not go out to real users then you put it behind a feature flag. If something needs to go out to users today, then you merge a change on top of your main branch that unconditionally shows the feature to users.
This way all code that ever makes it into production exists at a point along a single linear commit history, each merge commit of which was checked by CI and is the same history that devs developed against locally.
In my experience eliminating cherry picking makes a huge difference in reliability, and the only way to do that reliably is to constantly deploy to prod.
And yes, my current $day_job does this.
Long lived branches are a problem if you are doing hot fixes to prod. The goal is the process avoids hot fixes and critical bugs.
Some industries would not allow software to be released more than every few months. SQA needs to sign off. Users need to UAT. 2-week notice, 15-minute notice to log off. Downtime needs to be communicated to multiple time zones. Software versions need to be auditable for regulatory.
The only point I was trying to make is ‘publishing’ multiple times per day with a team of developers requires ‘devops’ and that release does not always mean to prod. The same setup for releasing to prod multiple times per day is the same as releasing to develop multiple times per day.
Long live DevOps
so then you either have to accept that sometimes it's broken and just wait on it, or otherwise agree that staging should always have a working copy of Kafka, except that means the Kafka team can now no longer use staging to stage Kafka changes, so then they have to setup their own separate staging environment and then plumb sufficiently representative test data into that new staging, and then and then and then.
When the tests fail, or the code quality goes down, the deployment fails. I would rather have DEV broken than PROD. Not sure how going directly to PROD would make anything better in this scenario. That is what Integration is for never broken always ready for promotion to prod. Plus a few broken dependencies means ‘yellow’ === impairment not unusable. If the app is unusable for any one dependency that is another issue.
Mocking data when you are ahead of another team is always the reality…
And devs should just not write bugs in their code, ever. Development mostly works, but is going to be broken in subtle ways that other teams are going to pull their hair out because their thing doesn't work, but their thing doesn't work because your team has broken development in a very subtle way that only that one team tickles. Your tests didn't catch it and the rest of your team didn't catch it during code review.
> except that means the Kafka team
That (a Kafka team, or a DB2 team) is a bit of a red flag for me. Many (but not all) “tech” teams like that are part of the problem. Cross functional delivery teams work much better, because running tech X in isolation is often not valuable.
The one exception to this is “x as a service” teams. Eg DBaaS teams, Or even a team that offers a pub/sub service. But in those cases such teams are literally measured by uptime SLOs, so if a pub/sub team can’t deliver uptime, they’d need to improve.
Dev is broken because devs are doing dev on it. I mean, it generally works, but it's the bleeding edge of development so there's no real guarantee that someone didn't push something that doesn't work in a way that the rest of the company is relying on.
What is the DBaaS or pub/sub team's commitment to uptime in the staging environment? It's staging. if they have to commit to a reasonable uptime, they can't actually use it as staging for themselves. Saying they need to improve is trying to handwave out the fact that they need a staging environment where they get to run experimental DBaaS or pub/sub things.
That all seemed to work reasonably well.
As a person who works in a heavily regulated field ($$$), all the audit / traceability requirements apply. We still move fast (multiple releases a day) because we have demonstrated that it reduces risk and promotes stability — something regulators are very interested in. In concrete terms, faster release rates are correlated with lower sevs (or incidents). But the underlying cause is really that faster release rates cause teams to adopt more reliable delivery practices, such as automated testing, gradual deployments, and so on.
UX-focused development. Your concern is mainly about user journeys, acquisition tunnels, etc. So you rely on user feedback a lot. So you ship, A/B test, iterate.
Also, if your service consists of lots of microservices, you don’t want to wait to ship them all at the same time.
The most fitting example to both is probably Netflix. I guess they ship to prod every day a lot.
Guess that is why Netfix is so Glitchy every other day j/k.
I should have kept that last sentence out but I will not edit it now as I asked….
Releasing multiple times a day in itself isn’t the point though.
The point is — adopting good practices to ensure trunk based development is pain-free. It’s about ensuring the release process is as automated as possible (if you’re releasing once a month, you are exercising your deployment process 12 times a year. When you’re exercising your deployment process 12 times a day, it really encourages you to work the kinks out.)
It’s also ensuring that your monitoring and alerting can keep up. Essentially it’s a signal to the entire org that “We will move fast. Deal with it.”
There’s data out there from even large non-FAANG orgs now about how this approach reduces sevs instead of increasing them.
I hated my DevOps role so pardon me for only reading the headings of the article, but talking about it in past sentences seemed very delusional.
That's not devops.
Devops is when you have developers who are empowered.
When you hire someone specifically to do devops you are hiring an ops team and calling it devops.
So, yeah. That still happens, sure. However, I do think it's changing these days:
linkedin > connections -> control-F -> 'devops' => 0 hits
"Site Reliability Engineer" -> 10 hits
I think hiring SRE makes sense. I think hiring 'platform engineers' makes sense. People still do that, and will, as far as I know, far into to the future.
...but I think hiring devops means you never understood, even vaguely, what devops was; you're just using a buzz word in your job ad.
DevOps is about people and culture. Your teams embrace automation, fail fast, lean — all the good stuff. DevOps is about concrete business outcomes. Not Ansible or bazel.
> hiring devops means you never understood, even vaguely, what devops was; you're just using a buzz word in your job ad.
Yes!
At that point, it sounds like you're just doing ops work.
That’s just half true. If something doesn’t work in production, guess who can get mysql production logs? Not the developers. Can you access production dbs (not that that’s the way to fix things, but I have seen that in many orgs)? Nop. And if there’s anything you can do yourself (like changing things via TF files), you’ll need to wait until the gatekeepers approve your request.
The gatekeeper are usually called platform engineers.
But in any case, I don’t want all that power (and responsibility) either if it does not come with an increased compensation.
I always got higher compensation in such companies/teams, because they weren't dying. At least, when I joined :) They needed efficiency instead of control and keeping salaries down.
A regurgitation of Sysadmins that knew some scripting?
Cloud Native Sysadmins?
Patrick Dubois said "a systems administrator working with an Agile mindset", and this guy started DevOps Days; which is where we get the term.
The "merged teams" spiel is from the "10+ Deploys a day" talk from Flickr, and it's but one definition of what devops means.
This is why I personally consider it failed, it's meaningless because it means different things to different people. You can never "do it wrong" if you don't even have a definition that everyone agrees on.
My first experiences had to do with the ability to add new services, monolith or not, and have their infrastructure be created/modified/removed in a environment/region in-specific way, and to be able to safely allow developers to self-service deploy as often as they want, with the expectation that there would be metrics available to observe the roll-out, and safely revert without manual intervention.
If you can't do this stuff, then you can't have a serious posture on financial cost, while also providing redundancy, security, or operating independently of one cloud provider, or one specific region/datacenter. Not without a lot of old school, manual, systems administrator work. DevOps hasn't gone away, it has become the standard.
A bunch of pet servers is not going to pass the appropriate audits.
From the article:
> adopt much more simple and easy-to-troubleshoot workflows like "a bash script that pulls a new container".
This style of thinking let's developers learn every nitty gritty pain of why we have frameworks. I see the same challenge with static site generators, they are all kinda annoying so people write their own, but then theirs is even worse.
I think a lot of Kubernetes hate is misplaced. It is a great piece of software engineering, well supported and runs everywhere. You certainly don't always need it but don't create a bunch of random bash scripts running all of the place instead of learning how to use it.
The whole ecosystem around it is an example of Conways law and and a Google product.
None of the people using it are google.
Google, also, runs its own hardware.
Shockingly it is a great product if you rent hardware, autoscaling is autospending. No one knows what a feature costs any more because its all just a big bucket your pouring money into for amazon to have 30 percent margin on.
We need operations people again, we need to stop using containers as bags for shitty "software" ... Do actual engineering.
Have worked with numerous cloud native engineers that do not have good foundational knowledge.
That's the problem. Yes, I'm the old man yelling at cloud, but indeed 'kids this days' don't know Linux, they know how to provision things from Terraform. In fact it feels like logging in via ssh and checking process with with `strace` is a lost art. Checking PCAP? That's a black magic!
No, I want to hire the grumpy, cynical greybeards, because they know their fundamentals, and they're immune to shiny bullshit.
There's no substitute for skill and experience, despite what the modern tech discourse says. "Anyone can program!" And yet, we have threads like this, where we find out, SHOCKER, that the best codebases are the ones maintained by like... four senior developers, half of whom contributed to several Internet RFCs, and half of whom used to work at Bell Labs.
I feel like we are through the looking glass.
You need to include checks when writing them so they actually are idempotent, just like with bash script, and they happily leak idempotency-breaking details like "restarting a service" works only if a) service currently runs b) because "restart" is a command, not expected state of "the service was restarted and is now running
I’m told this is gatekeeping. Yes, and…? I’ve always found it amusing that I see little to no anger being directed at kernel devs for their gatekeeping. Almost as if deep down, people know that they shouldn’t be fucking with the thing that runs the world unless they are actually good at it. Or maybe they’re just afraid of Torvalds.
It’s been a long time since I’ve had to do those things, so yes it’s becoming a lost art to me. However, on the rare occasion I need this sort of insight I do fine with JIT research.
Still in the same place they were 20 years ago. They’ve just been lost in the swamp of their peers that know nothing.
A lot of big dinosaur corporations are implementing it actively. Unfortunately VMs or Kubernetes or whatever tooling is still going to suck if you have shitty people using them.
This is such an inaccurate take.
>we need to stop using containers as bags
Containers and container orchestration are a NEEDED and REQUIRED piece of the technology stack in the current reality. That doesn't mean people need to be ignorant of the details that make them work.
As someone who's been around for a while we are in a better place than when we had "dedicated operations people" that just gatekept everything.
What you hate are teams just parroting what other people are doing with the tech, not the tech itself. But let me tell you, if you're going to have teams of people pretend to know what they're doing wrapping it in a standard "bag" sure does make it a hell of a lot fucking easier to unfuck when things go wrong.
If you're deploying NODEjs apps to the cloud, they sure are.
> if you're going to have teams of people pretend to know what they're doing wrapping it in a standard "bag" sure does make it a hell of a lot fucking easier to unfuck when things go wrong.
No one looks in the bag, they just roll back, the bag is opaque, the bag breeds more bags, less trnasparency and higher costs.
Step back and read what you wrote, if you described any other relationship in your life in these terms, your friends would be having an intervention. The codependent circle jerk has enabled so much bad behavior. The tooling you're advocating for is an enabler.
I want App Store like software roll outs. The infrastructure we have is kind of like that, if you don't look too close. We need to start doing the hard work to make it not suck. We need to stop pretending that this is a good place and put in the work to make it better.
Otherwise its never going to get untucked...
I don't use nodejs.
>No one looks in the bag
I sure have. Plenty of times.
>if you described any other relationship in your life in these terms, your friends would be having an intervention.
For having a standard tool that you know how to use? That doesn't seem correct and it certainly isn't proportional.
>I want App Store like software roll outs.
Okay it sounds like you want K8's.
That is not remotely true. Containers (and their orchestration) are a choice. Obviously people have their reasons for making that choice, but it is in no way a requirement. To say otherwise is either profound hyperbole or ignorance.
You'd need like 10 more Ciliums to make Kubernetes actually good, and they'll never all get adopted. The tech is a dead end that demonstrates some nice tech/patterns but got overhyped. It's like the Hadoop of containers
I like to think of kubernetes as a big orchestration platform that you can choose to use what you need. If an ingress and pods work then use that, otherwise extend an throw an operator up for what you need (it likely already exists).
Cilium for instance is great for that, so is Istio and the like. They aren't hard you just have to understand networking... which is nearly the same energy of running it on another orchestration tool or raw on a network device.
IMO you are thinking too much like an engineer if you are saying you “just have to understand networking”. Why? It’s always better when the problem gets solved in a way that allows you to not think of it too much. Right now SREs and the platform team do that for application developers because Kubernetes only does it halfway.
Infrastructure/devops/SRE is a pure means to an end of getting actual applications (the ultimate source of all the value in software) to run. It’s an obstacle. Right now the obstacle is bigger than it needs to be
Obstacle is big most of time because most applications are not easy to run. Most DevOps mostly came around because Devs flinging balls of mud over the wall and landing on the Ops side with a splat and then screaming when we can't build nets to catch the mud ball and Ops is covered with mud. Sure it failed just like DevSecOps fails because most of time, Devs don't care about anything other than closing Jira tickets and going home.
But companies don't want that, they want to support EKS and data centers, which means supporting the "implementation" side of all of the interfaces, which means getting down into the details.
The real problem is that every platform team, deep down, wants to rewrite EKS and they often do, which I would describe as "a giant money pit".
That will never happen, the only thing you can do is pay someone like heroku to take care of that for you. Or if your project is small and plain enough you run “serverless” which is just routing to another platform team, as I’m sure you’re familiar.
It’s complicated because there’s a lot that goes on, kubernetes or not.
In other words - Serverless is another platform team in the same way Linux is another OS team or Java is another language team. And for 99.9% of companies a language team or OS team would be absurd. There’s a big incentive for platform work to go the same way.
Things like having to remove finalizers appended onto resources when you delete them and they hang.
Damn near everything about Custom Resources.
This isn't a dig at Kubernetes; I love using it. However, I agree that it leaks implementation everywhere.
Most people in industry have never used Borg or something like it, and so think Kubernetes is like apex technology rather than a step backwards. And to most people working directly on tech like Linux/containers/low-level infra is something they don’t even consider doing because they subconsciously think it’s off limits. So to them Kubernetes is amazing rather than riddled with fixable problems.
The best part is multiple environments. I can run our full stack on my laptop with k3s and one call to make. We use the same yaml (with kustomize) for all three cloud environments.
It feels pretty boring, which is fantastic.
And the fact that many medium companies have “platform teams” configuring Kubernetes and gluing together basically the same set of tools (source, build, test, release, ops, obs) together in basically the same way is a huge smell that something better is needed. Basically, doing things the right way is actually a big operational/engineering/monetary burden for most companies that just want to write applications. And K8s is a big part of that
I do agree it could be better but it's Un opinioned nature also let's these companies shape it in a way that matches where they are. Also, probably biggest issues with it only come up if you are not in managed Kubernetes which many are.
So many arguments are based on strawmen...
I like devops / daily deploys, because they're part of the puzzle leading to higher quality code being deployed on production, and associated less stress.
The point is (for any individual developer) not to actually deploy their progress every day on prod, but to have the option to do so. This leads to code going on prod when it's ready, but no sooner. If the problem is more difficult than anticipated, code still sucks and needs refactoring, well, you're just going to work on it as long as it needs it and deploy it only then.
Meanwhile if you have let's say monthly releases, you will get the death marches, because delay of one day can mean delay of one month / quarter / whatever. Everyone feels the pressure to deliver, leading to suboptimal choices, bad code being approved etc.
As startups grow into enterprises, eventually there are benefits to be had from getting all the different SREs on the same page and working according to the same standard (e.g. compliance, security, FinOps...). Then, instead of each SRE building on top of the cloud provider directly, each SRE builds on top of the internal platform instead.
Hiring an entire team to build great dev-tooling and deployments, monitoring, application templates, org level dependency management etc is just too much to swallow for any medium sized or smaller business, so in that reality you wind up with a few heavily overworked devops folks who take up unhealthy habits to cope with the associated stress and risk.
In my 10 year career thus far none of the startups I worked for, even well capitalized ones had what this article, and myself, would consider to be a platform team. I only saw my first platform team when I stepped into a role at 6000+ person company.
It's effectively an underserved (and under-appreciated imo) area and responsible for a lot of pain and land-mine decisions companies make around their software product.
If you can afford to make the user the tester, you should. There is no moral hazard, only an economic one. If you have 5 million customers paying $1 / year, make the user do the testing via canary deployments, metrics, etc. If you have 5 customers each paying $1M / year, be sure to test it yourself.
The problem seems to be that people forget which regime they are operating in.
Er... No? If you take someone's money in exchange for goods and services, you have a moral duty to give them what you said you would. Not a broken version of it– what they bought. If you explicitly state they're getting an unstable product, then sure. If you actually do your best within reason and your service is broken, shit happens. Nobody is perfect, but you made a good faith effort to deliver on your promise. But if you don't, and deliberately don't bother checking if it actually works while happily pocketing people's cash, that's unambiguously negligent. Ripping off small groups of customers is no morally different than ripping off all of your customers— it's the same immorality at a smaller scale so you're more likely to get away with it.
I largely agree with you, except that you flipped the terms.
For leadership, the whole idea of "breaking down silos" is almost always lip-service, and to the extent that is/was a core mission of DevOps, it was always doomed. Responsibility without power doesn't work, so it's pointless unless the very top wants to see it happen. Strong CTOs with vision are pretty rare, and the reality is that the next tier of department heads from QA/Engineering/DataScience/Product are very often rivals for budgets and attention.
People that get to this level of management usually love building kingdoms, and see most things as zero-sum, so they are careful to never appear actually uncooperative but they also don't really want collaboration. Collaboration effectively increases accountability and spreads out power. If you're in the business of breaking down silos, almost everyone will be trying undermine you as soon as they think you're threatening them with any kind of oversight, regardless of how badly they know that they need process changes.
Anyway, the best devops people are usually excited to code themselves out of a job. To a large extent.. that's what has happened. We're out of the research phase of looking for approaches that work. For any specific problem in this domain we've mostly got tools that work well and scale well. The tools are documented, mature, and most even permit for a healthy choice amongst alternatives. The landscape of this tooling is generally hospitable, not what you'd call a desert or a jungle, and it's not as much of a moving target to learn the tech involved as it used to be.
Not saying every dev needs to be a Kubernetes admin.. but a dev refusing to learn anything about kubernetes in 2024 is starting to look more like a developer that doesn't know Linux command line basics. Beyond the basics, Platform teams are fine.. they are just the subset of people with previous DevOps titles that can actually write code, further weeding out the old-school DBAs / Sysadmins, bolstered by a few even stronger coders that are good with cloud APIs but don't understand ELBs / VPCs.
It doesn't mean CI/CD pipelines, Terraform, or YAML. Those are all incidental.
The moment specialised "DevOps" teams started springing up it was all over. We just reinvented the sysadmin.
1. I feel that one big and important aspect of devops that isn’t mentioned is that smaller releases are less likely to have killer bugs. If you can release one change a day rather than 100 changes a quarter then overall I think there’s a strong argument, not to be had here, that you’ll have faster releases and less bugs overall, assuming my next point. This doesn’t take away from the article, but it’s just something I don’t see discussed much.
2. I think a huge part of the problem is that business management keeps trying to abstract away engineering management. The most productive team I’ve ever been part of was when I was able to spend most of my time planning and coordinating the work, as part of an overall vision, while my peers did the implementation and gave me feedback. One side effect of this was that productivity was actually measurable. But the value of productivity is lost on business management who saw me as just engineer - one who had the authority, furthermore, to push back against stupidity and was therefore a pain in the ass. Technical management is not valued, because it’s not understood, and this is seen in the endless cycle of fads designed to make all engineers fungible.
I miss the days of having an expert sys admin and DBA to help with tough issues. Now we're all on our own with no training. Most of the good people got axed or thrown into other roles. Becoming good at something only to have it thrown away and undervalued pisses me off.
I brought CI/CD, automated deployments, and infra tests with me to what was a very "retro" non-cloud UNIX style environment. I left all the horseshit (kanban, retrospectives, scrum) for the others that are not really all that interested in actually getting shit done. While those guys had a bunch of shiny tools to play with all day long, they lacked basic UNIX skills that us old guys have, and you had to continually had to show them how it was done.
The results speak for themselves. Couldn't be happier (or more securely employed) at this stage of my life.
A person who competently knows all those topics _and_ how to write application code is worth their weight in gold
IFF the company consistently values those skills. IME, they’ll say upon interviewing or hiring that those aren’t necessary for their workload, until they suddenly are, where you’re lauded as a hero. Predictably, the memory of hero status fades when promo season comes around, because those “aren’t core skillsets,” or something similar.
Exactly what I'm going through right now with a potential PIP. Last year I was told there was a solid basis for me being the getting the highest rating, but only if my core work was faster. Now they want to PIP me like nothing else I contribute to matters.
“We’ve written down objective guidelines for what constitutes your relative performance.”
“Great, looks like I’m a shoe-in.”
“Well… there are intangibles.”
“Then why have the rubric?”
Bash alone has a million footguns. If you know to use shellcheck you can probably survive those, and if you read the bash manual in its entirety you’ll almost certainly be well on your way to greatness, but getting devs to read docs – let alone boring, Web1.0 docs, is stretch.
> basics of SQL…
Have you seen the knobs Postgres and MySQL have to turn? Do you know what they all do, when you should turn them, and by how much?
The documentation for both of these is enormous and highly detailed, yet in no way covers everything that can and will go wrong. You only find those things out by using it day in and day out.
As to SQL itself, “I can do inner joins” is about as complex as I’ve seen most devs do. Hell, mention a semijoin and you’ll get blank stares.
I think in general, you’re grossly overestimating the average developer’s knowledge breadth and desire to learn these things. They simply do not matter for most, because tooling exists such that most of the time, they don’t need to know. I consider myself decent at Linux and RDBMS, but that’s mostly because I genuinely enjoy both, play with them in my off-time, and have been running Linux in some form or fashion for the last 20+ years. Also, of course, my work specializations (SRE —> DBRE) have helped.
A lot of those guys just couldn’t handle a reality where they couldn’t name all their systems with cute Dr Who hostnames and do all their work as local root with riced out bash prompts
Devops at least forced them to the same table with the dev org, and if they ended up getting replaced in the transition then I shed zero tears
God riddance to those guys. They thrived in silos.
They are not worried about understanding what they do, or improving their practices, they are worried about status and how others perceive them.
Those people may be able to protect their status in the short term, maybe even bluff and be promoted. But they will never truly understand what's going on.
Worked with so many of these guys. They really are the worst.
I really enjoy working in a deploy often and fast environment though and I firmly believe that fast feedback loops are one of the most important things for development speed. And this is what DevOps at its heart is about. How you achieve this and how reasonable it is for your situation is left for you to decide.
But every single idea I read in that post is just wrong. Like author never worked in siloed team where you had to wait blocked for a week so DBA guy picks up your change request. Then if something went wrong on prod you had to wait for SysAdmin to basically be your typist because you did not have access right.
It is not that you don’t need DBA or SysAdmin but for devops purpose they are assigned to a team - which makes companies needing more of those people NOT LESS - because earlier you had single DBA to know all of company projects which was cheaper for business. Now idea is you have people in the teams so you don’t throw stuff over the wall but single team can deploy and operate their project with full knowledge.
Well of course there are companies that take 5 jr devs and now assign them to be devops team but that is company work organization problem not devops problem.
So that even if you're building small website for your local soccer club it's probably run through GHA on every change with a full red/green deploy process, run on autoscaling groups and so on.
Never mind that most of these applications' databases could fit into RAM on a single server with 24 cores and never even touch the system limits.
The industry has been given servers with 100x the resources and collectively said "naw I'm good". Can't wrap my head around it.
Where devops went wrong in a lot of teams though is assuming it's a full time role for a specialist that then does your devops. That's not devops. It's ops. And these aren't developers but operations people. Embedding them in teams is still progress though as it removes obstacles.
But if you do it right, this is not a full time thing at all. The wrong way is generating a lot of busywork for your devops people to develop loads of yaml files that feed into things like Kubernetes, Terraform and then enable organizations to codify their structure into their deployment architecture using microservices (Conway's law). I'd suggest not doing that and doing things that minimize the need for devops people. Like using monoliths.
I prefer solutions that minimize my time involvement. I use monoliths so I don't have to babysit a gazillion deployment scripts. I need just one of those. And since I don't have micro services, I use docker compose, not Kubernetes. The deployment script is just a few lines of bash that restarts docker compose. It kicks in with a simple Github action. The amount of time setting that up is a few hours at best. I rarely need to touch those files. We have no Terraform because our production environment got created manually and we're not in the habit of destroying and recreating that a lot since we launched it years ago. And it's simple enough that I can click a new one together in an hour or so. Automating one off things like that has very low value to me.
- the application has hardcoded paths.
- the service discovery isn’t dynamic
- the branching strategy doesn’t account for edge cases.
- the build process doesn’t account for edge cases.
- and many other things that are related to bad practices.
I recall an old boss saying he wanted stable dev environments which sounded an oximoron. I’ve always aimed to have an environment where I can reproduce a desired behavior wether is a faulty or not.
Cloud-specific saas (like proprietary databases and AWS lambda) are awful for this though, and we avoid them where possible. Our S3 code was a bug farm until I discovered minio. Now we’re stuck with Snowflake because scale and speed, but the fact that I can’t iterate fast locally has caused me to lose days of my life that I’ll never get back. Hoping I can hack something together with duckdb when I get time.
Jesus this. No one knows where the money goes. If you can't tell me cost per customer, per user then your business is missing key metrics.
> ... "discovered" that troubleshooting Kubernetes was a bit like Warhammer 40k Adeptus Mechanicus waving incense in front of machines they didn't understand in the hopes that it would make the problem go away.
Wackamole with problems...
The part where he talks about the death of QA.. yea. This is enshitifcation in action.
> Jesus this. No one knows where the money goes. If you can't tell me cost per customer, per user then your business is missing key metrics.
This is where something like FinOps comes into play as a framework to adapt to your needs.
It’s all straight out of “programmers are also human” https://youtu.be/aWfYxg-Ypm4
One of my favorites in my first couple of months in my career was testing a service that accepted user uploads and returned some value based on the upload. I thought “oh, I should just test a butt-load of invalid files”. Where am I gonna get a butt-load of invalid files for our service? I’ll just write a script that iterates and uploads everything under my `C:\Windows` folder. Discovered 2 bugs. One was that some random binary files there caused the parser the service was using to segfault and another about how the service stored those temp files before validating them. The first required involvement from a totally different team in the company because of “how serious it was”. As a fresh college hire at the time I got a ton of praise for “exposing such a critical buffer overflow” that went undetected for years (lol, I had no idea what I was doing but I took the praise). The latter was because the developer of the service never cleared their /tmp directory where they wrote user uploads before validating them. I ended up filling the VM disk with junk that took down the entire service.
I'm so glad you quoted that because I missed it skimming the article and it's the funniest shit I've ever heard.
I'm not familiar with the reference but logs, events and metrics seem pretty useful. Port forward, shell into containers, view logs, etc. I don't see what is so bad about it.
The problem here is that once your in and need tooling what do you do? Your likely going to have to re-build the whole container with that tooling on board (if its even possible)...
Containers aren't problematic in themselves. There are reasons to use them. Ruby, php, JS python (fucking venvs) have this habit of contaminating a system. Containers are a great way of... containing that shit.
But the moment I'm giving you a single file binary.. why in the name of all that is holy are you putting it in a container? If it's untrusted software then there is a good reason but then why are you running it at all?
And that's the thing. Depending on what I ask you to do there are places where your not going to say "let me put this in a container" ... Most of that is "good software" and "performant" ... Its a statement about everything that comes in that "container"
Ability to inject a container with debug tooling into specific pod.
Of course, if you have one file (or many) that you run on one server and downtime is ok during updates then fine to just do everything manually. You could even keep the code on that server as well. ssh in, edit compile and run your production app directly from the out directory. Basically run production on your dev machine. It sounds pretty silly / "non professional", but there are times where ultra low cycle times outweigh other things (like when you have one developer and desperately trying to win your first customer - you know "do things that don't scale"). The main thing is, you need to know the trade-offs you are making.
You can even checkout your code, edit in vim and re-deploy. Crazy talk for production of course, but if you need it, it is possible. Basically it is not that far off from running a binary on a server if you need it to be.
It doesn't scale very well: the larger the codebase/team, the more burden on each individual to make this work.
I used to have passwords for everything and could deploy things and get things done on a dime, now there are layers of bureaucracy and middle fingers everywhere I turn.
Is that DevOps?
I briefly worked at a place where the SREs were responsible for replacing displays on the factory floor and the devops engineers took rotations approving terraform PRs all day long.
As it turns out, a lot of orgs just rebrand their job postings with whatever fancy industry term gets applicants, then they rugpull you into very mundane ops chores
Both are solutions to big but specific problems.
(DevOps was a push against crazy corporate IT, where developers handed over the sacred tomes of operational manual and the blessed JAR/WAR files and IT took over from there. And operations was on-call, and ... had absolutely zero fucking idea what to do, the manuals were fake, and they had no expertise, and the whole handover was just burning money, etc. So dev teams got access, and things got a bit more programmatic anyway as scale kept growing. From the early days of Capistrano and Puppet to immutable images running on EC2 and nowadays to CI/CD with containers running wherever.
Microservies was kind of an organic step from that. Both to handle resource scaling and mostly to handle product/project coordination. Extremely limited scopes helps local reasoning to make okayish decisions most of the time.
See also Rust's ergonomics principles about "reasoning footprint" and the Bounded Context concept from DDD.
Of course the trick is that, it looks easy when someone gives a talk at a conference showcasing their company's a-ma-zing whiteboard-to-yet-another-webservice tempo, but it needs well-funded, good-faith constructive/supportive security and platform teams. And in practice it's famously hard to keep these complex interconnected systems/projects/orgs on track. And even when you have that many companies then try to force things [the good old hammer-nail anti-pattern] https://www.youtube.com/watch?v=PAew2jhr2zs )
This is what is we with DevOps now. The short lived dev teams had it over to the next dev team with basically no documentation. DevOps was just another stop on the shitty process parade that Agile contributed to - all about speed, forget docs, who has time for real tests, and for requirements we'll just use JIRA tickets the devs work on.
Surprise, surprise... Our bug and rework stories take about 25% of our capacity now.
Institutions and institutional corporations generally can’t make the switch in mindset to multidisciplinary service teams, and established project managers will even fight it tooth and nail since it looks like a threat to their jobs.
Microservices are orthogonal. Architecturally they are just SOA, now with json/grpc instead of wsdl et al. Operationally and strategically they are slightly a better fit for service teams, but only at scale. Small and medium enterprises will experience inefficiency instead.
Conway’s Law is super applicable to understanding these dynamics.
Having been a frontend guy some 10+ years ago, into a network engineer, then infrastructure engineering and now SRE. The amount of people on both sides of the developer circle and operations circle that do not want to understand what's going on is mind boggling.
I was around when VMs were hot, when treating them as long living pets was just toil that operations dealt with. The collection of shell scripts to make that toil go away was nice. Then puppet, ansible and the like.
Now we are in the golden ages of Kubernetes and orchestration platforms. We have a set of standards for how things can be operated. The terms are obfuscated sure, but the core concepts are still the same underneath the abstraction.
I agree that platform engineering is a good place to be, and honestly it needs to be understood more by all parties including executives. They were bought and sold cloud on the idea that it's all managed, but that cannot be further from the truth, wrinkles will show as scale grows and your use cases progress in any environment, at home or in the cloud.
Unfortunately good platform teams often aren't seen. A good platform just works, metrics just exist, logs just work, tracing just works out of the box. Things don't often go down. It's really only visible when things fail. If you do a great job implementing a self service platform you're often met with executives wondering why you're there because the cloud does it all!
Applications are highly visible to all, but so are the layers underneath and they all work together if done correctly, I wish that was more understood.
For context, I'm currently running multiple environments of Kubernetes, on premise and in cloud. Our team prides itself on using open source solutions utilizing the operator model. Prometheus, Thanos, Loki, Tempo, Istio, Cert-Manager, Strimzi Kafka, Flink operator, Otel collector etc. We do billions of requests a month and TBs of bandwidth with microservices. Have at a minimum 4 9's of uptime, and our cost footprint is extremely small. This comes from a 4 man platform team that also handles on call for all applications, security, cloud budget, and operations. It's not impossible.
I guess I can't emphasize enough that understanding what the orchestration systems, the tooling and the stack are trying to do makes everything easier. As a developer you can understand your constraints and limitations. You can build off of known barriers. As an operations or platform engineer you can build things that don't require constant babysitting or toil.. you can save hundreds of thousands of dollars not offloading your observability to data dog or the like, you can make an impact. The technology is already here.
I've debated writing it up and posting it somewhere, maybe I should. There's so many ways of doing things now that it's quite overwhelming sometimes.
K8s is a complex tech that requires multidisciplinary experience that small and medium orgs cannot afford. Even if they could, there simply is not enough talent to hire. My own experience shows that k8s makes developers less productive, because running a heavy stack locally is not exactly conducive to fast development cycles. I don't feel empowered, I feel abandoned and left dealing with a steaming pile of shite that used to be the responsibility of a DBA, Ops, and security. Unfortunately, the trend for hiring "full stack developers" who can do frontend, backend, infra, and DBA aka. "I want a whole team for the price of a junior dev" is not going away.
This guy and the person that quit the bullshit industrial complex 6 months ago should get together and launch a startup.
Better yet, we should all go and join Jeremy Howard's answer.ai pro bono. Besides being miraculously headed by a guy who is Not An Asshole, it incidentally also had the most refreshing launch post (in the warm and fuzzy way) this side of the AI bubble.
The launch post concluded with this heading:We Don’t Really Know What We’re Doing.[0]
I mean, for the finest minds in our respective fields, what else is there left to say really?
No joke, containers are amazing, regardless of how quickly you try to move or how often you need to deploy.
I remember a project where the performance turned out to be horrible because someone was running Oracle JDK 8 instead of OpenJDK 8 and that was enough to result in a huge discrepancy, here's an example of the request processing times during load tests: https://blog.kronis.dev/images/j/d/k/-/t/jdk-testing-compari...
That would have been solved by Ansible or something like it, of course, but containers get rid of that risk altogether, since you need to package the JDK your app needs (and that it will be tested on).
With a bit of work, using containers can be quite consistent and manageable - have Ansible or something similar set up your nodes that will run the containers, run a Docker Swarm, Hashicorp Nomad or Kubernetes cluster (K3s is great) that's more or less vanilla, something like Portainer or Rancher for easier management, Skywalking or one of those OpenTelemetry solutions for tracing and observability, throw in some uptime monitoring tools like Uptime Kuma, maybe even something like Zabbix or a more modern alternative for node monitoring and alerting and you're set. Anything that's self-hostable and doesn't tie you up with restrictive licenses (this also applies to using PostgreSQL or MariaDB instead of something like Oracle, if you can).
You don't need to have every team branch out into completely different tools because those are the new hotness, you don't need to run everything on PaaS/SaaS platforms when IaaS is enough, realistically most of what you need can be stored in a Git repo that will contain a pretty clear history of why things have been changed and even some Wiki pages and/or ADRs that explain how you've gotten here.
The situations in the article feel very much like corporate not caring and teams not talking to one another and having no coordination, or growing to a scale where direct communication no longer works yet not having anything in place to address that. If you're at that point, you should be able to throw money and human-years of work at the problem until it disappears, provided that people who hold the bag actually care.
For what it's worth, regardless of the tech you use or the scale you're at, you can still have someone in charge of the platform (or a team, where applicable), you can still have a DBA or a sysadmin, if you recognize their skills as important and needed.