Infrastructure as code is not the answer (2022)

The questions are always: is it better than what we had before? Are the "problems" common to other tools? Are there better alternatives? This article doesn't address any of these.

> I heard a story once of an engineer running a “terraform destroy” on his laptop — in the wrong tab. He was in the Prod directory — and starting nuking production. Realizing his mistake, he Control-C’d out of it, but it was too late.

That's the stupidest argument I have seen. If you do 'rm rf /' or 'DROP TABLE' in production, nothing can save you.

I'll say straight away, IaC for things like AWS in my company at least is invaluable.. the thought of even trying to deploy services with user policies etc, let alone the mess people would make of that is a disaster.

Terraform done right with a provisioning account in the build system to deploy the IAC... It's brilliant. Hands down. Brilliant.

All of the downsides are 10x worse when doing infra manually.

Just do IaC well to avoid the pitfalls, rather than polishing the turd of manual infra.

With the same logic you can say that contraceptives are not the answer because they aren't 100% effective.

Of cause IAC is mis-sold like anything new. It's not the end of all your problems. But as someone who's had to deal with the opposite too, infrastructure-as-a-weird-mystery-that-accidentally-happened-over-20-years-and-only-long-bearded-john-knows, IAC is fantastic.

All of the things mentioned in the article seem more a result of bad IaC, instead of general IaC...

> Packages go out of date. Features are deprecated and stop working. Code syntax changes. Images are no longer available.

This is true for all code and we have tried and tested methods/principles for dealing with that.

> Someone’s user account got built in, and they don’t work here any more.

No comment needed.

> He was in the Prod directory — and starting nuking production.

Say it with me: Elevate Permissions

Letting teams self-serve XaaS is how you get 50 dramatically mis-sized and ill-configured DB instances in prod. This team thought they’d have 1000 QPS, and have an instance that’s 50x as large as necessary. This team doesn’t know what IOPS are but figured they’d better have a lot of them, so they’re paying thousands per month for an io2 drive when the entire dataset fits comfortably into memory. This team has no idea (fair – there are a million knobs) how to configure Postgres, and the defaults suck, so they’re getting 1/4 the performance that they’re paying for.

IaC can be done poorly, just like anything, but at least if someone knows the underlying tech that they’re modularizing, downstream consumers won’t each have to relearn it.

A lot of people work in tech these days. A lot of people write blogs.

Anyway... back to scale, consistency, and speed - IAC.

Infrastructure as Code is fine, its that the current implementations are too whacky and too manpower-intensive to make it worth it.

Good automation would be where you need less manpower afterwards..

IAC is one technology I will defend to the hilt. IAC is difficult because the underlying problem it is trying to solve is hard. Different people over time are making updates to a mission-critical, stateful, system that need to be coordinated with each other.

IAC is complicated, tedious, poorly documented, risky to deploy, and never perfectly aligns with what is actually going on in production but it is still better; a lot better. Manual infrastructure is also all of these things except you have nothing to grab onto if you are trying to understand an environment and what is safe to change.

If your app is very simple and you can use an out of the box solution, you should avoid IAC. But the moment you want to do something custom you need it.

To avoid data loss in prod, you best be using

    lifecycle {
        prevent_destroy = true
    }

https://developer.hashicorp.com/terraform/tutorials/state/re...

> trying to get some old module to work in a new cloud environment where a bunch of assumptions about how things are done no longer apply

Square hole type energy. Just because you can coerce the code to run doesn't mean it's a good idea.

New environment, new assumptions, new code. Copy-paste what's relevant, but in a new project.

> getting totally confused because what they see in the code doesn’t match reality

Terraform compares state to detect drift? When reality doesn't match code, you clobber reality or you update the code.

> It Prevents Drift!

> This is true if and only if, you only ever, ever, ever, create code to change things in your environment.

It at least makes drift obvious, even if it requires attention for it to be addressed.

> code is, in my opinion, really not good at being documentation

Unlike comments, IaC "can't" be inaccurate after applying.

> But in my experience, the effort to keep your code up-to-date is a sink for engineering time.

If the code is out of date, it's because you made a click change and didn't update the code. When this happens, terraform tells you exactly what changed in simple scenarios, and in more complex scenarios you can use import blocks to rein things in.

> Yes it’s a clickbait headline. You clicked on it, didn’t you?

Got me good. The article is a good launchboard for discussion.

Overall, the IaC scene has plenty of room for growth. Terraform import blocks and code generation is helping reduce the problem of "I clicked in the portal now I gotta update the code" because you can sync both ways. Code updates infra, infra can be used to generate code.

I read the addendum. I think it was a good, understandable rant.

When I had to invent a complex HA-solution including infra for an ERP product I went with Ansible, and yeah, fine, it worked, but it was also really hard to figure out a good design for the orchestration. YAML is awful, I actually hacked up a decrepit YAML editor for the specific usecase to not have to deal with formatting details and whitespace myself.

If I had stayed there I would probably have designed an XSD/XML to YAML pipeline to have an easier time structuring configuration and allowing quick edits to certain values in the Ansible scripts. I.e. reinventing a GUI administration of the code.

I've also been on OpenShift, and time spent on fiddling with details in YAML on the team was a lot. The tradeoff was fine, it allowed us to run test stacks with several Oracle databases and a swarm of supporting applications locally with relative ease.

But I don't find the IAC way to be some kind of messiah, or really have much novelty. We've always had more or less arbitrary DSL:s for common remote operations, right? Sometimes in bash, sometimes in PHP, sometimes INI, sometimes XML. Sysadmin is a mess, and you should say thanks to your dedicated sysadmins every day for putting up with it.

Besides the point, but Medium.com is not the answer if you want to publish text on the internet.

As for the article he's not proposing anything better.

Most of the gripes he has against IAC have nothing to do with the fact that your configuration is in a YAML file versus a GUI. They have to do with the problems of the systems that configuration is touching. Moving to a GUI would not help dependency hell, for example.

I disagree with a lot of this, but here are the bit i really disagree with:

> The second issue is that code is, in my opinion, really not good at being documentation. I have written code myself, and come back to it six months later, and been totally baffled by what I was doing there.

This might be true of python, node, java, any language really where you fiddle with complex structure and algorithm. It is not true about IAS.

Quick example: I hate yaml with a passion since forever. I also never ever heard of ansible before, but 7 years ago, i was starting a new job where most of the deployments would be with ansible. All 30+ clients infrastructure were extremely simple to understand to me, even weird, locked stuff with NFS mount and chroot everywhere across 5 commodity servers. I could help any client sending a mail "X does not work, is it on your side?" within 10 minutes despite never having touched the infrastructure or ansible before, and having trouble believing yaml (despite my preference, one positive about yaml: its very readable). Terraform might be sightly more complex due to trickeries, but its basically the same: we have an intern, the terraform part was really the quickest bit of our deployment he understood.

> It’s not something that a non-engineer can easily read, or even an engineer who is not familiar with Terraform or whatever can easily read.

Again, hard disagree. Our manager, who isn't an engineer, learned how to read terraform in three one-hour sessions. He's not good enough to understands the specifics, but he can now review new modules with us. Why do we have new modules? We have 3 network engineers, not developers, who started doing terraform. They learned in two sessions (i think only one of them needed the second one tbh). They are publishing two new modules next week. We started showing tf a month ago by the way. We have a really good engineer who was really good explaining everything, which help, but it isn't hard.

IaC is undeniably a good pattern for dealing with complex systems. The bigger problem is that it's not a solution for delineating responsibilities between software engineers and infrastructure engineers. All the specifics about an environment that need to make their way from software->infra (and in some cases the other way) still need to be worked out, and that's often a bigger problem than getting code to a box that always has access to a database (oversimplification).

I tried to outline some of these ideas in a post [0], full disclosure I work for a company solving this problem. Our approach is to treat IaC as _part of_ a broader abstraction for applications.

1. https://noop.dev/blog/build-apps-not-infrastructure/

I've done IaC is various forms over a decade and I believe the term does more harm than good.

It's not "Infrastructure", and it's not really "Code" either. "Infrastructure Configuration as Text" perhaps would be more descriptive. Or just stick with the samewhat older term "Configuration Management".

These days, however, it's not really in vogue. Nobody thinks they need it anymore. They have Kubernetes and containers now. When you ask enough questions about how those are managed, it usually turns out some state is being managed somewhere outside of version control. I believe the pendulum will swing back in time, but in the meantime there will be K8s on my resume, not IaC.

A significant benefit of IaC is the work-savings it yields for recovery from a total-site-loss event, or the migration to multi-geo/multi-site.

Sure, very few IaC environments are going to make that sort of thing seamless--bootstrapping new infra from nothing in a hurry is always going to turn up every skeleton in every closet and require a lot of manual work. But having even 80% IaC automation yields huge time savings in those rare-but-impactful "start from zero" cases.

Having even just the easy/copy-paste parts of the infra encoded in automation can be the difference between hours and days for disaster recovery, or days and weeks for creating a new geo/staging/whatever environment.

Back in the olden days, I had a junior dev ask me if he could learn ops, I said sure, I was just a hacker myself. He wrote a script to make sure the hard drives in all the servers were good. A month or so later we rebooted a server and it wouldn't come back up. We finally rebooted another to see if we could learn something about the dead server. But it also wouldn't boot. Then to our horror we learned that the disk check script was writing and reading to the boot sector to make sure it was good. All of our servers were affected. Good times.

Setting up solutions in AWS is a most painful experience. Even AWS consultants do not find it easy and trivial.

However, IaS being a complicated hornets' nest is more direct evidence of vast complexity in architecture.

The best way to address it is to decrease the complexity of the system. It will have great rewards across the board.

Most systems composed these days, have far more moving parts than what is needed.

Simplify and shrink.

Some projects really do need it, no doubt about that, but I predict that somewhere around 80% at least coud make do with far less, and the project will benefit from it to to bottom.

Imho, I think we've gone a bit overboard with Terraform at my company. Sure, it'll automate a bunch of stuff, but only a handful of people will really "get" it.

I think the rest of us who interact with it don't really fully understand what's going on. We've basically traded one set of scripts only a few folks understood (bash, perl) for Terraform in this case.

I totally see the benefits, but sometimes I feel like we try to force everything into it, even when it might not be the best fit.

Almost all the listed downsides of IaC are also downsides of _any_ code. I was starting to address the points one-by-one but that’s the gist of the entire article.

The alternative is we all start using Alice to click and drag pieces of logic together to build software. This is the equivalent of building infra through a web UI.

I don’t see the author’s point. Sure IaC isn’t a magical solution with zero problems but was anybody claim that?

reproducible - IAC is a quantum leap over manual on this one.

self-documenting, visible - with iac you have the config in writing and in a repo vs. what the admin wrote in a paper notebook

prevents mistakes - maybe. but at least the mistakes are documented and visible.

prevents toil, lowers costs - iac wins as soon as you factor in time saved on set ups, deployments, etc... versus manual... not even close

I'm going to agree - because there's also the complexity of keeping all the state in sync too.

Give me Fly.io or something similar any day. I'll pay the premium gratefully.

Regardless of what the author himself is saying, this is a clear example of someone that did not actually get passed intermediate level in knowledge and expertise.

Infrastructure as code can be useful for complex systems.

These days, the cloud providers have pretty good abstractions that IMO diminish the need for IaC for many use cases.

We also went from - click around in the gui -> automate everything -> click around in the gui. Zero regrets.

We went even further than that and ditched kubernetes, and containers altogether and moved everything to just EC2/elastic beanstalk.

Took us 2 months to write everything in terraform initially. Took 2 days to shift everything back and maintenance is just so much easier for us. Much easier to find people that can manage infra this way too.

To this day I struggle to find a tangible benefit to the old stack. Everything is just “better” now and half the cost.

YMMV of course.

The big risk is that you'll outgrow the GUIs and end up with what amounts to code but all the logic is in YAML files instead.

IaC, like DevOps, is a mindset.

Dumb article, written by someone who probably has never had to setup dev>acceptance>prd environments.

GL clicking your way through that buddy.

Honestly a lot of the arguments in that post seem in favor of IaC, not against it.

Truly cursed take. Loved the “uh… ha ha, got’em!” walk-back in the addendum. Here’s an another anecdote for you, if that’s what we’re basing our arguments on:

My prior team had dozens of services to deploy and they were all done in the console. It took hours to roll out deployments, people constantly fucked something up (wrong version, missed dependency, config variable, etc), you had spend sometimes up to an hour to get a picture of how a service was currently stood up, was almost completely opaque and thus extremely expensive to get new hires up to speed, obviated the ability to perform post deploy automation for smoke tests and such, required giving leads broad access to environments that needed to be locked down, and the best part was you had to do it all over again for each environment. All of this went completely away when I rolled it into CloudFormation, hooked it into CI/CD pipeline, and let a maintainer kick it off with the click of a button. I guess I must have been doing something wrong /s