The DevOps Engineer's Handbook

> Focusing on defining clear processes for recurring events and building the fundamental building blocks that allows you to automate when it's absolutely needed should be the method, not spending more time writing Terraform.

For one, I don't really consider terraform "automation," and more IaC, but I'll digress - this is all well and good in mature organizations with robust processes and aligned leadership. In practice, however, and what I find most often, is you will find very small "devops" shops in companies that aren't necessarily "tech" sized 50-300 people with a devops team of 3-5 people (if they're lucky) that the organization, or sometimes even themselves, see as glorified IT sysadmins. They're always seen as an expense, usually critically understaffed, and if you leave teams like this to decide on their own what is "necessary" to automate you're going to get weird/misaligned/dysfunctional results, and even moreso if you let the business decide this, which is what usually happens, and they don't really give a crap if some poor former-sysadmin has to spend 12 hours a day clicking buttons in aws console as long as they get what they needed (actually have seen a guy making 150k to basically do just this).

So what happens, like I said, is teams get into this hell-loop of manual task after manual task, which not only requires large amounts of mental bandwidth to keep track of or keep up to date all the documentation or playbooks surrounding these manual tasks (if you're lucky to get even that), you have to deal with the inevitable mistakes and errors that are common when doing things strictly manually, which eats up a ton of unnecessary time and thus $$.

I agree though most devops consultants are terrible, and the industry is driven by this, however, this is the specific niche I've carved out for myself, coming in after big terrible crappy consultant that basically just pitches a brittle jenkins CI setup and some basic terraform and charges you $250k for their time. I actually really enjoy doing it too, and the challenges and issues are almost always unique to the org, even if the patterns are similar - so it's always interesting.

So, long story short, unless you have a super robust process and mature system, it's usually just a lot easier to default to "automate" and come up with reasonable exceptions when it doesn't make sense to do so, rather than the other way around.

Git ops (declarative infra and config) and containers, make automation really, really easy, and completely eliminate all sorts of classes of problems. We push code to prod dozens of times a day without any issues, for months and months at a time. Typically backed by only a single devops engineer to keep everything humming along or building new automation. The automation is the clear process, spitting out messages regularly to email or group chat somewhere. And provides the audit trail.

I worked at a traditional finance company and we had a team of 8 people in traditional operations and another 30 people doing manual testing around the clock to support about 20 developers, 10 network staff, plus another 20-30 managers or leads and security. We could only deploy once a week and there were always issues with "final check out" on sunday morning when hotfixes had to go in or config was modified.

True. That's why IT Services companies have such massive practices dedicated to DevOps. Its a great annuity business for them.

I completely agree, not every task deserves to be automated

> not spending more time writing Terraform.

Also, there's another bit of nuance to that, as well as your overarching point about "automation isn't free," in that writing Terraform/Tofu isn't usually the long pole in that tent: debugging the raging PoS most certainly is (along with its associated https://xkcd.com/303/ of waiting for the "plan, attempt apply, puke, goto 1" loop)

And, in almost the exact same vein: writing any automation carries with it two downstream bits of work: monitoring the automation and having enough context to debug it when (WHEN) it falls over

I probably made it sound fancier than it is, but in a consulting situation by the time someone is willing to listen to what I say they've likely been experiencing considerable pain and are willing to take advice, so it's a little different than when you are stuck there. which is basically, biting off the smallest pieces you can that give the largest value you can (to me, the value I use to measure this is roughly by time spent per week estimates). So if it takes me 20 mins to automate a task that is spending 3 hours a week of devops time, of course this is a super easy target. Not all tasks are like this and people are bad at estimating how much time they spend on stuff like this, so it requires a diligent approach and honest introspection (one of the hardest parts to me, people are wildly delusional sometimes). This involves a lot of meetings, alignment, etc.

The time I've been stuck in these situations, it's mostly about inflicting or bringing notice to enough pain that the business backs off on some tighter deadlines to give more time to automate the tasks that will free up the most bandwidth or time - and it's a lot of small bites. I typically will start (if I can) by insisting anything new that makes sense to be automated (making a basic estimate of time per week, and effort to automate, also something you need to factor in is long term maintenance to use as a pitch to sell to management) and stick rigidly to that until it starts taking over other legacy processes in sort of a slow strangler pattern. This can look wildly different depending on the infrastructure setup and needs and how much firefighting you're doing - which there is usually a lot of. So sometimes that first step is just putting out all the fires or gaining enough visibility/monitoring to ensure you know what fires need to immediately be jumped on and which ones don't, and most importantly, proving that to the business.

Unfortunately though, leadership buy in (to me) is the hardest part almost always which is why I say "inflicting pain" the way I do, it sounds bad, but if they do not feel pain you will never, ever get priority to do anything, because IME like I said "devops" guys to most businesses (and even a lot of technical people) are glorified sysadmins that cost and demand way too much.