Terraform is not the golden hammer

Pulumi has dynamic providers. You define the crud operations and Pulumi manage the state.

I've used https://github.com/Mastercard/terraform-provider-restapi successfully with a cloud provider which provides a suitable HTTP API. There was a bit of fiddling with JSON formatting and their API docs, but it wasn't too hard all in all.

But like you say - now I've done that, I want to do it for every UI that I'm forced to log in to!

This is how I feel about any kind of configuration after moving all my personal systems to NixOS.

“What do you mean run an installer and update these files…”

IMO, this is an area where I think Terraform + Ansible pairs so well together.

If there’s ever a gap in what Terraform offers you can pretty easily fill it with Ansible.

I wish I could use it for everything as well. Every other tool the business depends on just scares me to shit that it's config isn't in code and we can't have a proper backup of it. At least Datadog supports terraform for most things and not only we manage all our infra through terraform we manage all our monitoring with it too. I doubt very much I'll ever go back to non-monitoring-as-code if that's even a term.

Infrastructure, all the monitoring as well as all the on-call rotation configurations (and anything else that is in that loop) should all be in code, and all changes should be reviewed the same way as application code does. If it doesn't, you can't really trust you're gonna be alerted properly when things start breaking.

I wish I could use it for personal things too, I'd rather have my bank account settings, my government tax information, yada yada in a personal terraform repository for example. Change of address? Commit a a change, check if the plan is good and apply to change it everywhere. Though having lots of experience with Terraform I can only imagine what the equivalent of trying to delete an S3 bucket that still has data in it is for a bank account.

Your intuition is right on pulumi. Creating these kinds of extensions is minimal effort. You can start out by adding a "dynamic resource" class to your infra codebase and extract it into a plugin later, or not.

These are not the same as "real" pulumi providers that run across all supported programming languages, but I think they are a good enough fit for the cases you mention.

This stuck out to me too. Terraform absolutely does check the current reality of the state and applies changes to do what the HCL tells is to. They are either using terraform in a really weird way, or this article was written by someone that doesn't actually run terraform themselves.

I also agree, but I don't have experience with the providers he is using.

When working with AWS it will always reapply the terraform configuration.

A great use of this is for account hardening. You can run it daily to make sure it's configured correctly.

Relatedly, I'm pretty sure the earlier statement

> For some resources like RDS or EKS, it won't check if the resource already exists or not. So if it's missing, nothing is going to happen as it's marked are deployed in the tfstate file

is also wrong.

The author also seems to think DSL means "descriptive languages" and that Helm "even" supports kubernetes, when in fact it's only for that technology.

The stage when terraform read the reality to compare it to the current state is called "refresh". It can optionally be skipped.

There's also brokenness around terraform for_each and providers. If you have a module that creates a Kubernetes cluster and then applies a helm chart to it, you can't convert it to one that takes a bunch of cluster definitions that it iterates over using for_each. Basically, there is no way to do this in a date driven way. Sucks.

There’s also a refresh flag on plan. It’s always worth using before your apply CI step.

As the sibling commenter notes, Terraform has a refresh flag, but I wonder if Kubernetes’ model is better here. Rather than a one-off process that tries to update everything, Kubernetes has many small controllers which are essentially processes on the cluster that just run a control loop. Each controller corresponds to one resource type, so it will just loop over incoming events that pertain to the resource type in question and attempt to reconcile the state of the target resource instance with the desired state. If it fails initially, it will retry with back off. If something doesn’t stabilize after several minutes, an alert can notify a human.

The key differences between the controller approach and the IaC approach are, I think, lots of little processes continuously reconciling state for all resources of a given type (many small loops that touch all resources of a given type on the entire cluster) versus a one-off process that tries to touch just the resources it cares about and if it fails it just gives up.

One thing Kubernetes definitely improves upon Terraform is that Kubernetes uses a YAML “assembly language” for its infra as code, but that YAML could be generated by a real programming language. Terraform expects you to write HCL, which is an accidentally re-invented programming language (every IaC tool provider thought static configs like YAML would suffice as a human interface, but as they gradually realized the need for more dynamism, Terraform and others would bolt on one dynamic feature after another until they had a slow, unfamiliar, and counterintuitive programming language). Terraform has a CDK that allows writing in other languages, but I’m skeptical that it liberated you from Terraform’s model of the world (e.g., if I rename a variable in CDK, does it try to destroy and recreate the underlying resource as with Terraform?). I’m also concerned that rather than allowing us to generate YAML in the obvious way, it will require bizarre inheritance patterns like the AWS CDK. I would be curious to hear from folks who have used the CDK.

The upside though is that you're adapting that known universe to a way of working that makes sense for terraform users.

I read you. The half of the Terraform's value today is "let's have common aesthetics - especially the examples and snake_case_naming_convention - and convert any REST API in the world to that".

The typical example is to start from there:

https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_V...

and arrive here:

https://registry.terraform.io/providers/hashicorp/aws/latest...

If you review it carefully, it is apparent how much coding effort and many moving parts were used to perform a transformation which seems disproportionately primitive.

I’m partial to Pulumi for this reason. It allows devs to use familiar languages to define infrastructure with their familiar tools, write tests, and even interop with existing terraform.

I think this is less that it is opinionated but more that the HCL evaluation feels like a pile of hacks. There are unclear rules on what can be evaluated when and what dependencies are possible. Part of this is so that `plan` can work as it does, but it seems like there are just major gaps in general. For example providers can't depend on resources. This makes it very difficult to for example set up EKS then use the kubernetes provider to manage the resources in the cluster. The solution is obviously separate stacks but that brings in a whole bunch of other problems.

I think Terraform is quite possibly the best tool available, but there are clear flaws with both the model and the implementation. I think if I were to make a Terraform v2 I would make `plan` completely pure. This would avoid the provider issues, make validation and testing in CI easier and a whole bunch of other benefits. Of course there are downsides. For example EC2 instance IDs are random so you can't just include them in your pure plan. You would need some type of placeholder that is used for evaluation. This does cause some issues as it limits the operations that you can do with that value (so you can't pick the instance size based on the random instance ID) but overall I don't think it would be a major issue if the final substitution was handled well by the framework.

You have to understand that when IaC was new, the marketing was “it’s so simple you can just write YAML/JSON/etc” because frankly the industry was too dumb to understand that “using a real programming language to generate a description of the desired resource state” and “using a real programming language to imperatively reconcile the current and desired states oneself” are different things. So Terraform began with something that resembled YAML in its static-ness, and over time, more power was required so they would bolt on a dynamic feature but were reluctant to give the impression that they were building a programming language so the feature would be as obscure as possible. But that wouldn’t be enough either so they would add still more dynamic features, each comparably obscure until in time they’d built a complete, obscure programming language.

But this wasn’t just Terraform! The entire industry did this too. CloudFormation began as simple JSON, but over time they allowed you to encode the abstract syntax tree of a shitty programming language in your YAML, and CloudFormation would interpret it. However stupid that may sound, in the Kubernetes world, we have Helm which lets you generate YAML with text templates which is honestly the dumbest idea in the world (imagine a compiler that generates syntactically invalid machine code if the input program has an extra white space character).

Of course in all of these cases the answer is staring us in the face: use a static language (YAML, JSON, etc) to describe the desired state, and use a higher level language (like Python or Starlark or Dhall or etc) to generate that static desired state description. The only thing Terraform (or any IaC tool) should care about is the YAML description. That it is generated from Starlark or TypeScript is just an implementation detail.

Instead of that, though, we get CDKs which are so close, but admittedly I haven’t used them in anger yet.

What new cloud features was not available in Terraform for months?

It usually takes 1-2 years for AWS to roll out their latest updates to the regions I use and by then Terraform is stable.

> But using cloud APIs directly was great for us

This is fine, I've done it extensively myself for some of the bleeding-edge cloud stuff, but the importance of things like tracking state, managing hierarchical resource dependencies, or retry/back-off logic shouldn't be tossed aside simply because there are gaps in what's available in the Terraform providers. Especially where change management is important (basically any enterprise company).

I'd caution others reading this against abandoning something altogether and writing bespoke IaC tooling simply because the stable approach doesn't cover every (bleeding) edge case.

You'll spend a lot of time reinventing the wheel, and while it's fine for certain situations (like when you only care about desired state, not known state, for instance), you'll move faster (and likely safer) by sticking with tools like Terraform for the bulk of your infra, and augmenting here there with cloud APIs/SDKs when needed.

Same here. It's incredible how much efforts developers are willing to put on popular "devops" tools when the job could be done faster with 200 lines of Python.

Would the time spent re-implementing a specialised Terraform subset be better spent simply maintaining a private branch of the AWS Provider? You can add your secret/special API without having to do all the other heavy lifting as well.

This makes your own effort for customisation minimal, keeps your knowledge portable and because your added features can be separated in to different files and the provider API is stable you can also easily backport/fast-forward new changes.

The emphasis on stateless here is that your desired state us described in code that resides in your repository. Actual state is what you have in your cloud. No need to spend time on state format, storage and related logic and complexity

This is my preference as well. I've done everything from makefiles and bash scripts to a monolith Go program that statelessly provisions/tears down resources.

Even makefiles are pretty straightforward, though you really want operations to only trigger when checksums differ -- timestamps result in a lot of redundant operations. As long as everything is idempotent, it's pretty straightforward.

> Terraform modules are always a few steps being from the upstream public cloud offerings.

My experience has been the exact opposite. Usually Terraform offers support for cloud services long before the vendor provides an SDK or supports it with their own offering (e.g. Cloudformation). There are still dozens of AWS services, for example that have no CF support offered by AWS.

I mostly agree with this, but you're not considering the time it takes to make and maintain it over time. It is non-negligible and has to be in the balance as well. But from a pure technical POV, I agree.

If Terraform was used for the deployment of the infrastructure, then state IS the actual configuration of the system.

All that a plan does is evaluate what is going to change in the current Terraform state by performing a dry-run of the Terraform code that you have supplied.

If you would actually like to make changes to the Terraform state based on what the Terraform code evaluated then you run a Terraform apply - which will, for the resources deployed via Terraform, update the configurations themselves and update the Terraform state by using the Terraform code as the instruction set.

You can actually see this in action with plan and apply as the output will show you +,-, and ~ where ~ is settings that are going to change but are not new configurations or configuration to be removed.

Edit: Learned from some other comments that Terraform has a 'refresh' command that will take deploy+n time configurations done outside of Terraform and sync those configurations with the state. This might be what you ideally are looking for after deployments?

It's not true, and what you describe is correct.

I can only assume that the author has used some provider that hasn't implemented this properly (helm, I believe, is one example), or that they've run into one of the cases where terraform treats configuration/attachments to a resource as a separate resource (e.g. IAM role vs attached/inline IAM policies).

The console command [1], mixed with local variables, is quite handy for debugging locally.

1. https://www.terraform.io/docs/cli/commands/console.html

I agree most of your comment but the cloud-agnostic part.

Heck, I never get how terraform can be cloud-agnostic... If everybody thinks having a same language (HCL) is equalivent to cloud agnostic, YAML exists...

It is literrally impossible to create a simple VM in 2 different cloud providers without defining them twice with their own specific parameters.

If you use AWS provider, resources start with aws_, if it's GCP it starts with gcp_ and so on. It is not possible to have a "resource vm { name = ... provider = aws }"

> Terraform is great for "fire and forget" deployments.

Indeed, terraform is great for exactly the opposite: For making sure both the initial infrastructure and subsequent modifications to it are first proposed, discussed, reviewed in code and applied afterwards instead of anyone with privileges tinkering with settings on an "as needed" basis in whatever console thereby ending up with an infrastructure where you have no idea who turned on the frobnicator, why it was set to 11, and what might be the consequences of changing the setting.

If you just use a cloud provider's UI you will have no separation between desired and actual states. Then, whatever people change it to is the only truth.

The hard part with auto-importing is the resource id. This is usually generated server-side and is not included in the hcl. Often resources define a user-supplied identifier as well, such as a name property, and this could be used for auto-importing if the property has unique constraints applied against it. However, not all resources have this feature, so its not a universal solution.

Hm, maybe what we really need is a new entrant in hyperscale clouds that is built from the ground up for IaC and just does away with the split between state and reality. I would love to see one, anyway.

What's the point of reimporting? Aren't you then no longer separating desired state from actual?

In the article, the problems the author discusses are:

1. Inconsistent behaviour between providers. e.g. if a resource has been destroyed since the last `terraform apply`, then some resources/providers would recreate the resource, others wouldn't. (Similarly, there's not a guarantee that the state after running `terraform apply` matches up with what's there, if the provider is happy with its state file).

2. The dependencies of already-applied resources can block `terraform apply` if the upstream API for these resources suffers an outage.

3. If a `terraform apply` applies some resources before failing, this can result in an inconsistent state. Either the resources need to be deleted, or imported.

I'm not familiar with Pulumi; what aspects of these would Pulumi help with?

Wrong thread?

Ah, congrats on being part of today's ten thousand.

I very highly recommend investigating it more and trying it a bit. Terraform isn't mere cloud deployment.

As a small project you can start by deploying an EC2, RDS and some cloudflare records to go with them, all linked together with terraform. This will give you an initial idea of its capabilities.

> learned about (yet another) cloud deployment tool

Please don't read it as an attack, not the intent. Amazing that a HN regular can "learn about Terraform" only in the latter part of 2021!

What does the AWA CDK do differently than use public APIs?

Hmm, as a former AWS employee who has used both heavily, my experience has been the opposite.

Terraform’s AWS provider calls the APIs directly, whereas CDK generates Cloudformation, an abstraction on top of the AWS APIs. For me, using Terraform was significantly faster than applying the same stack via CDK.

Or do you mean you’re able to iterate faster writing CDK vs TF?

CDK synth is reasonably fast but CF is really slow

I'd rather devote my time learning an agnostic tool like terraform. I can be part of your team right now working on AWS but tomorrow I might be working for an Azure shop.

From my experience, terraform is almost always slow because it's making API calls out to the cloud providers, and a lot of that in turn is slow because many providers offer "eventually consistent" which terraform needs to compensate for by doing roundtrips to validate that changes are have become visible (applys failing because of that was a common problem in the early days of terraform).