A more mature take on stateless Terraform
bejarano.io
bejarano.io
Everything else is indeed optional, but the fact that a useful three-way comparison can be done at every run is why terraform works so well and why other IaC tooling is a failure when it comes to change detection and cleaning up.
If it helps, you can think of your terraform files as HEAD and the state file as HEAD~1
Imagine two branches trying to update staging using gitops
But if we just take the GitOps branches problem, they are indeed not handled, but that's because you generally configure your GitOps pipeline to only allow interaction from a specific ref, not any branch you might have lying around ;-) The SVN version would either be trunk-based deployments or revision-based, where you have to tell your reconciler which revision you want.
For those that don't know, the way you delete a resource in Terraform is to remove reference to it in your code.
So for example to create an EC2 instance, I make an EC2 resource block (or use a module that itself references one). Then run Terraform. Terraform looks at see that this ec2 instance isn't in the state, but it is in the code, so that must mean it is new and I need to create it. So it creates it and then adds the metadata about that resource into the statefile.
Now to delete it, I simply delete the resource block from my code and run terraform again. Terraform looks at the state file and sees that there is supposed to be an EC2 resource, but doesn't see it in the code. So it then deletes it from the cloud provider and then deletes it from the statefile. That's how you delete.
So if you remove statefile, then how does Terraform know that you deleted something? It isn't in the code so it doesn't even "think" to check on it. You need the statefile to remind Terraform that it used to exist. Again, creating an object is easier because you have it defined in the code and so Terraform can try to reconcile it with an existing resource in the cloud provider, and if it can't reconcile a pre-existing resource than it can assume to create one. But if you delete it without a statefile, then Terraform can't know it exists. The author seems to know this, which is why they suggest using Git to check the last time the code was run, see that there used to be a resource block and that it has been deleted. Then Terraform can conclude that it should delete that resource from the cloud provider.
My problem with this is that in order for Terraform to work you must preserve your git history. Which does generally happen but in enterprise environments we have had to do some nasty voodoo with git on occassion, and I fear that this could significantly mess up my cloud infrastructure as a result.
Trust me, I have spent a lot of my time reconciling and repairing statefiles over my time as an SRE. I know tf state very intimately. It is a hell of a lot easier to repair a statefile (which is just json), than it is to repair a git commit history. And I live in a world where you have to assume that the day will come that you need to perform this task. When that day comes, I will be happier repairing a json state file than git commit history.
Three examples, but first context:
You create some .tf files that create a bucket on s3 on AWS, and then create an object inside the bucket. This creates a graph data structure that describes the AWS provider, the S3 bucket and the S3 Object. You then apply this desired state, which causes terraform to create some resources in AWS and then describe the state as it was last known in the state file.
Scenario 1: You delete the object from AWS but don't tell Terraform about it. You re-run apply, terraform notices that it is gone and re-creates it, so far so good. Even Ansible and SaltStack could do this.
Scenario 2: You delete the object from the desired state in your .tf files. You re-run apply and terraform notices that you used to have an S3 Object in your desired state (because your state file says so), but since it is no longer there it goes to talk to the AWS API to see if the object is still there and if it is, it will delete it (or, plans to delete it and then it is up to you to apply or discard that plan). This is not possible if you don't record your previous intent. One-way tools fail here.
Scenario 3: you make a mess where you remove the object from your .tf files and also manually edit the state file and remove the object there as well. Terraform now knows nothing about what the world used to look like. But you then decide that the S3 bucket can be removed and since it's all managed resources anyway (no dirty clickOps tricks in the console) you tell Terraform to destroy the bucket. Terraform doesn't see any dependencies, and asks aws to nuke the bucket. But since the bucket isn't empty, you get an error. Terraform doesn't know anything about any objects in the bucket, and thus it couldn't clean it up.
Now, the scenarios here are oversimplified, and you would probably not setup an IaC tool to manage 1 bucket and 1 object if that is all you'll ever configure. But when you start creating multiple environments that each need their own configuration drift control and controlled resource destruction you really can't function without it.
In general, I like that approach better, because it just uses the state that already exists. It would be kind of nice if Terraform had a mode where it avoided creating extra state where possible.
In Kubernetes, the state reconciliation happens by controllers based on information in etcd, in a way, etcd is what the statefile in terraform is. The apiserver is what the 'current' state is, including metadata and event fields, and any on-disk manifests you might have would be the desired state. The only big difference between the Kubernetes reconciler and terraform is that event-driven nature. Terraform tends to be a series of one-shots where Kubernetes (the controller manager and controllers) is a constant stream of reconciliation events.
The overarching theme in both is 'desired state reconciliation via declarative configuration'.
As for client-side apply and server-side apply in K8S, you're essentially having the same deal as terraform: you feed it a manifest of what you want, and it will figure out if it is new, old, needs to be updated or needs to be destroyed.
"the desired state, the last known state and the current state"
Why do you need to know the last known state. You have the desired state and the current state, you run your code and you reconcile the current state to the desired state
I get why this is impossible/expensive to get the current state with AWS, which doesn't expose a simple "show state" API, but managing a state store is extra work and thus extra fragility.
However ideally I'd like to list credentials and providers to manage, then say "manage all resources with some form of tag "managedByTerraform=12345" (this could even be in the resource name -- for my fortigate management my scripts manage firewall rules, addresses, etc which start with AUTO_, and ignore the other objects)
It would then run, generate a lock resource, find the current state, compare with desired state, create/delete elements to reconcile actuality with desire, implement, free the lock
Then all I need to do is have that human readable file and I can get back to where I am
If I lose the state, or it gets corrupt, I'm in for a world of hurt with terraform.
Trying to do it without state has been tried plenty of times, and it's bad. (Chef, SaltStack, Ansible, CFengine, Puppet to name a few)
It only works somewhat okay if you're simply writing to a single managed unit, i.e. a file. You know what all the contents of the file have to be, so you can replace it wholesale. This does not work for a graph of resources that interact with each other.
Separately, this also doesn't work with a shared responsibility model and doesn't work with deleted code.
Terraform has a state because that is required to solve the problem in a reasonable manner. It doesn't have state just because it was thought to be a fun exercise ;-) If the problem being solved is not _your_ problem, it is highly likely that using terraform at all is also not going to help you.
> If I lose the state, or it gets corrupt, I'm in for a world of hurt with terraform.
And that's why you secure and version your state. I.e. using S3 and versioning and a restrictive policy.
Additional potential issues: how are you generating your primary keys before resource creation? Probably by some hashing of attributes known at creation time? And how are you avoiding collisions? How are you dealing with modules that may have gone through multiple commits/versions since the last time you ran an apply?
There's just too many places where you really want to have a mapping of at least logical IDs -> physical IDs to ensure consistency. The one big advantage CloudFormation has here is that it just handles the state for you, but there's definitely still state.
If Terraform were truly declarative, all of your infrastructure is represented in code and you don't need state because the only thing it needs to compare with is what's actually there.
However, cloud APIs generally suck, and getting rid of state universally would be almost impossible for other reasons.
Think of the footguns you're introducing if they designed it the way you suggest. What happens when you pass the provider configuration an assume role that you typo or forget to edit correctly but it's still a valid role your user credentials actually have permission to assume? It then deletes every resource that role has the power to delete because you nothing for it defined in that particular root module. Safety is more important than purity or ergonomics or devex here. Heck, that may be the core difference between ops and dev.
You would see the deletions in the plan and bail out.
This is much much more difficult than it sounds if you want to be able to handle any amount of refactoring or repo reorganization and not be cut off from being able to handle resources from "before" a certain time.
How terraform or provider upgrades would work is another minefield. It's not just a matter of having access to the old terraform source, it would need to evaluate it with the same terraform version and providers - so those would have to be acquired and run within the state backend.
And then this is still assuming the previous run that you're trying to emulate was applied 100% successfully...
Good catch!
Recent versions of terraform have come out with a `moved`[1] block, but it is really unintelligent. It is a step in the right direction, but I can't use variables with it or string interpolation or anything.
Trying to get the state to be happy has been very difficult.
I prefer ansible for this because it doesn't use state[2], but as my company uses terraform I am left with the unfortunate need to manage state.
The best thing that I've been able to find to help in this fight is a little script I created to help automatically make moved blocks[3]. In a refactor you typically want to take resource blocks from the main.tf file and move them into their own module. You first moved the blocks into the module, and then before you edit anything you run that script and it matches up the blocks that were moved using Levenshtein distance. This approach may seem hacky but it works very well. Then I just take the output of the script and append it to my tf file.
I found the article refreshing and poignant. I will definitely be hoping for innovations like this to finally improve The state of the art because now that OpenToFu[4] is on the scene features like this that might chew into the bottom line of HashiCorp will start finally start being looked at seriously.
1: https://developer.hashicorp.com/terraform/language/modules/d...
2: https://blog.djha.skin/blog/that-does-it-ansible-wins/
3: https://gist.github.com/djhaskin987/b856c157bcfce6d2e4ee19f9...
The addition of `moved` is welcome but I feel like it was a huge mistake to use the FS as the module system such that moving a directory or renaming it is a breaking change, despite the fact that the name of your resource never changed.
That design decision has shot even the most experienced engineers in the foot once they see a renamed directory killing a database and spinning a new one up.
This is why we read the `plan` output before approving an `apply`. Terraform is not causing this outage, humans are.
So instead of generating state by doing API requests, this would generate state from code in git, which does not necessarily represent the real world state of resources.
As much as I'd want Terraform without state I believe using Terraform code in git as state creates more problems than it solves. This approach isn't able to recognize changes made to resources done outside of Terraform. It also assumes the previous version of the Terraform code in git is always the one which got applied last, which isn't necessarily the case.
What you're talking about here is unrelated to whether the state comes from regular Terraform state or if it's generated using the source code, as the refresh phase should happen during both.
When everything stands out, nothing does.
Sorry for the off-topic comment, but I do think this is valuable feedback for the author.
Sidenote: TIL about the <mark> HTML tag.
I feel like imperative Terraform would be easier to reason about, since it matches up with the non-terraform workflow (i.e. go here add this, then go here and configure that, yadda yadda).
When I read HCL, I really struggle in understanding the order of priorities. Like I know that surely some things must be built first, and supposedly it's all handled under the hood. But I'd rather be in control tbh.
Then again, as I said, I don't have much experience with Terraform so maybe I'm way off?
Yes. HCL is the easiest part of terraform.
State management / knowing what a particular tf apply is going to do and if it will put your stuff in an irreversible state where you have to go in and manually fix it is the hardest part. For us, we are forever struggling with IAM policies and KMS key aliases, especially if there is also a developer boundary coming into play as well.
1. One resource depends on another through the attributes, like an autoascaling group depending on a subnet. This is handled automatically by Terraform - updates/ordering will happen in the right way.
2. Dependencies that exist outside of configuration. For example some domain name should be created before some service starts. You can add those explicitly through `depends_on` for the resource that needs it.
Going imperative would lose most of what's interesting about Terraform. Imagine you create a service then change some attributes. Now you want another environment that looks the same - how would you do that imperatively? Replay all changes from scratch? Flatten the history and create a new script? With declarative Terraform what you're running is already in your configuration, no changes needed.
Instead of trying to gotcha me, do you want to provide some examples where creating a resource and changing an attribute results in something different than creating one with the final attribute values?
S3 buckets are another - it’s no longer even possible to create some types of buckets that exist in the wild today, thanks to changes in how canned ACLs may be provisioned. Older buckets have their access rules unchanged however.
These are two super common resources in the biggest cloud. I’m sure if I went digging in Azure or GCP I could find dozens of non-trivial cases of this that actually make a difference - database-as-a-service offers are likely a target-rich environment for this.
With this in place, and ideally with allowing folks to define custom backends using the plugin interface (decoupled from core, versioned) this story could make a ton of sense.
Disclosure: I’m on the steering committee of OpenTofu.
There are many cases I would accept simple hashes vs client side encryption, too. Everyone is going to have different needs but it makes no sense HC ignored making this better for so long.
In Terraform’s defense, it’s often the fault of providers and/or APIs.
This would work the same as state locking works now. You can apply an extra provider for state encryption/decryption just like you do for state locking/unlocking.
It's already been requested (with pull requests) for Terraform for a while now and Hashicorp keeps rejecting it, presumably because it would undermine features of Terraform enterprise.
OpenTofu is already considering implementing this feature as a result.
This means that if you have a CD pipeline (triggered itself by a commit) that does `terraform deploy`, you'd need to have your CD pipeline do commit _again_ to save the changes to the TF state files.
On top of that, you lose the ability to re-run a previous CD pipeline to rollback to a previous version of your infrastructure.
The goal of Git is just to store the lifecycle of a piece of software, imo. It's not the reponsibility of the repo to know the state of the deployment.
But I agree that the gain from that is not crazy.
If the answer is "you should never branch and merge state files", then the corollary is "you should never put state files in source control".
Versioning your state is a good idea and is already easy to do. For example if you use the S3 backend you can enable object versioning on your bucket and then everytime the state is updated it is versioning and you can rollback state that way. It is a good practice. Similarly, you could use the local state backend, which just keeps state in a json file in your project and then you could commit that json file to version control. Now git is versioning your state.
But the author is not suggesting this, they are suggesting to not have a state file at all. You look at previous commits to determine state. So for example if you create a new resource for an ec2 instance in your code and run terraform apply, how does Terraform know it is new and needs to create this ec2 instance as opposed to updating an existing one? Normally it would look at the statefile and if it isn't in there, then tf knows to create it, if it is already in state then it knows to update it and reconcile the metadata in state to the declarative state in the code. But without a statefile this can't be done. So the author looks in the git history at the previous commit. If you can find the resource block in the commit history you know what it was last set to. If it doesn't exist in the history but exists in the current file, then it is presumed that the resource is new. If the resource was in the previous history but is different in the current commit then it is presumed that those changes need to be applied.
I'm sure this works in simple use-cases. But at any moderate scale this could break down very quickly.
The author wants to eliminate state, so they suggest using git because as it turns out... you do need state.
But the author didn't eliminate state at all, they just eliminated a seperate statefile. If that's what bothered them, then sure, the problem is solved. But now you are using version control as state and those don't always line up. If you only work linearly then it is perfect. But what happens when you need to start tracking a pre-existing resource? You have to import it into git history somehow.
Back to your point, you really should not be modifying git history at all. There are tools to do so, but anytime you modify git history it is technically a mis-use of git. There are side-effects of doing so, especially when working on a large distributed team.
Basically the author trades one minor problem for many major problems.
I think the author misunderstood the problem (maybe because it's not crystal clear on Hashicorp's website): it's not that not all cloud providers provide tags, the problem is that only certain percentage of resources are taggable.
So imagine you are using your stateless TF code based on tags only: you could only deal with those resources that support tags. Whereas now, after configuring the backend on S3 with DynamoDB, it's basically "fire and forget" - you can use practically any resource. Paradoxically, TF supports even more actions on resources on AWS than CloudFormation.
So maybe for some simple setups like EC2 with S3 this could work somehow, but for anything more complex the current implementation is far superior.
> For any given already-existing resource, Terraform needs to know how, in the absence of its code, determine its dependents so they can be destroyed in the right order.
should be
> how, in the absence of its code, to* determine its dependents so they can be destroyed in the right order.
How can you express this without state? Or more generally, any “random_password”, “random_choice” etc, or even anything that is a “write only” property
You can’t. So this is dead in the water.
Do you in practice really use random names? In my experience, I'd just use a loop vm01...vm10 for the names and the passwords aren't needed to identify an instance after the deployment so here randomness isn't an issue.
Random passwords, write-only attributes (like database master passwords) are the most common.
How do you express “create a DB with this strong password, then put it in a s3 object”, then later “actually put it in SSM rather than s3”?
With Bicep, we mostly deploy only the initial state and then we either re-deploy the whole thing or, if this isn't possible due to the interruptions this causes, add migration scripts in an imperative language (az cli/ pwsh). Which is admittedly the much less elegant approach.
They haven't built it, because it's really really hard to do.
And the author’s mental stretch of claiming git is the state is still a bit of a leap.
If you want to see this, just make a terraform apply. After it completes, go change something directly in your cloud provider and run another plan. The top section of your plan will highlight the changes that Terraform found remotely that don't match existing state.
So all this would do is remove known state, so you could do a 2-way reconciliation instead of a 3-way reconciliation. You would compare current-state with declared state. This is how Kubernetes works. But to be fair, Kubernetes actually works in a slightly more controlled environment than Terraform.
What an absolutely bone-headed idea. That gives you the worst of both worlds (I guess they’d argue it’s the best instead).
I guess that’s my mistake for assuming Terraform wouldn’t have an extra reason for me to hate it without making something up.
They diligently recorded all secrets in AWS Secrets Manager. They used IAM roles and policies to control who can see which environment’s secrets. Great!
But then I discovered all the secrets in PLAIN TEXT dumped into the state file?! And this is known behavior that Hashicorp defends as reasonable. They suggest you just encrypt the state file. So the next time there’s a big oil spill, let’s just throw a big blanket over it and call it a day.
Terraform’s solution swiftly obliterates all the audit, key rotation, separation of duties built into secrets manager. Indeed, what is the point of using Secrets Manager at all if you use Terraform?
Next, they recorded state in a hit repo. So all those secrets are committed to the repo. So now how am I supposed to encrypt the state file? What a disaster.
But wait. There’s more. With the team growing, how are we supposed to manage shared resources? Do I have to run tf apply, wait for completion, and then immediately commit and push, and hope I don’t have to manually manage a state file merge conflict? Or should I use some bizarre, self-built mutex?
Ugh. Terraform is the worst. I was always unimpressed by its feature set and documentation. Now I hate it. I don’t understand why it is so popular which there are such better alternatives.
In one project, we had Terraform run from GitLab CI. The CI was creating a plan. The reviewer had to approve applying that plan.
I'm curious about your usage of Ansible for AWS resources. How do you delete them? Do you have a policy of always having 'state: absent' for at least one commit?
terraform init --backend=gcs --bucket="xxx" --prefix="my-deployment-name" --encryption_key="my-random-bits"
Terraform (or CloudFormation, or Pulumi, or Crossplane, for that matter) shine when you need to create resources. Think of the state as the inventory of what you've created (or imported).
Having 3 different sources of truth (what is, in AWS, what should be, in the .tf, and something else -- in the statefile) can mean nasty 3 way merges, which i
But I don't manage thousands of different resources, I manage 50. It feels to me that the overhead needed to manage thousand struggles to scale down without bringing all the required baggage. It feels like kubernetes vs docker-compose.
That said, the concept of using an S3 bucked for storing state I saw elsewhere in these comments is an interesting idea so I may revisit terraform.
Even so, git only exacerbated the problem of secrets being in state files.
Would you please elaborate about the alternatives (beside Ansible which you’ve already mentioned)?
PS: Infrastructure Manager was released just 1 month ago, so probably not.
This avoids storing the secret's value in the state file (it's stored as an empty string) and also keeps the terraform plan clean.
The usual practice is to keep the terraform state in an encrypted S3 bucket tucked away in a separate account (CI/CD, management or similar), with IAM policies controlling who can actually access the terraform state file in a cross-account setup. Limited access to the bucket with the terraform state can be controlled via the S3 bucket access IAM policy. Typically, there is an overarching, cross-account organisational IAM role that controls such access.
Each terraform project typically gets its own dedicated state bucket. Sharing the same S3 bucket for multiple solutions is unusual.
The encrypted S3 bucket persisting the terraform state file has to have the S3 versioning enabled. If one stores the terraform state in git, they are cooking it wrong.
Static and third party provided secrets are stored manually in the Parameter Store and are sourced via «data» blocks in terraform programmatically. Access to the secrets is controlled via the appropriate IAM roles and policies. Access to the secrets by humans is by exemption that is attached to a (typically) SSO role associated with an access group in the organisation's own IDP. This is no different from the non-AWS secret management solutions and tools.
Ephemeral or «low value»[0] secrets in the Secrets Manager can be rotated daily (e.g. every 24 hours) – to discourage the manual access to, storage of and reliance on them as well as to encourage to retrieve the secrets programmatically.
terraform is a vehicle to get from point A to point B, and is not an AI nor a substitute for the knowledge of the platform.
The terraform documentation is excellent, by the way. It is no replacement for the knowledge of AWS though.
[0] Generated via the «random» provider.
Ansible is fine to be clear, for managing things like configuration within compute instances and databases (such as user accounts for example). It is good for this, but for building and maintaining the creation and lifecycle of the infrastructure itself, it is the wrong tool for the job.
First of all, you are using local state in Terraform which is a terrible, terrible idea. You are already seeing why based on your problems with it. I've never seen an organization that actually works on a team with local state backends. You need to use a remote state backend. The most popular is to just use a provider like S3 (or equivilant with other clouds). The state file is pulled from there at plan/apply-time and then pushed back up there when complete. Then everyone is always getting the latest statefile and it is shared automatically without needing to commit it.
This solves other problems too. Like what if two people try to apply at the same time? Well that is why state-locking exists. Before the remote state file is pulled down, it is "checked out" and locked. Now only that person can use the state file until it is checked back in and unlocked. THis happens seamlessly in Terraform once you set up a state lock backend. You still just run `terraform apply` and all this happens seamlessly. Now if someone else tries to do it while you're already updating things, then their cli will wait until you are complete. THen they will end up pulling down all of the changes you just made. If they were running the same code as you, then it would say there is nothing to update. Easy.
Encryption of your state file also happens this way, by encrypting the remote backend. It doesn't solve local encryption and this is admittedly a problem with Terraform which Hashicorp has refused to address, but is already being addressed with OpenTofu. But as a result, secrets should simply not be in your state file. You should use a provider like secrets manager to do this for you and then you only have references to your secrets in state, instead of the secrets itself. This is simply a known rule just like you don't commit anything secret to a git repo, you don't put anything secret in your terraform. It's the same way.
Lastly, state versioning. This is accomplished through your remote backend state provider too. We use S3 at work, so we get native versioning there and we can (and have) rolled back state in emergencies (although this itself is not a great practice, just like how you should never edit previous commits in git), but it can be done and preserved this way as a safety mechanism.
As for your auditing and key rotation of secrets, it once again sounds like you are doing this wrong. You shouldn't be updating the secret keys themselves in Terraform. You should only be creating the secrets and passing around references to them in Terraform. This requires a better understanding of how Secrets Manager works. Creating a secret in AWS for example only creates an empty "repo" for a secret. It has an id and metadata, but no inherit value. This is what you do in Terraform. Adding the value to the secret is a seperate (hopefully automated) process. The secret value itself should be hidden from honestly all of your users (even yourself). Rotation should happen automatically with Lambdas or automated jobs of some kind. You could even do something with ansible for this. You rotate the secret value, but the secret reference doesn't change and everything should continue to work with secret versioning and some proper architecture on your side. This should NOT be done in terraform. Auditing your secrets again, is not a job of Terraform. This is a seperate process from IaC. Terraform is IaC. Use the right tool for the job.
State merge conflicts should never happen as you fear because it is wholly managed by the terraform binary. Yes you might have to resolve state conflicts where someone added too much drift outside of Terraform and you need to fix it, but the cli provides tools for making these changes and you shouldn't edit them yourself. Using these tools (like `terraform state mv` or `terraform state rm` or `terraform import`) should resolve state conflict without causing any sort of merge conflict. The state file (with proper state locking) will only ever be edited by the tf binary and only by one person at a time.
So I wouldn't hate on Terraform just yet. It sounds like you and your whole team is using it wrong.
However, these issues are just a few of the most recent reasons I dislike TF. I’ve never been impressed by TF documentation. It always comes up lacking for me. Subjective, I know.
Similarly subjectively, when I was looking around about 6 years ago, TF had way less coverage of AWS resources than Ansible.
Basically, my experience is that the level of thought and quality of engineering that went into Terraform is way less than Ansible. And I am annoyed because somehow Teraform won in spite of that. My hope is Terraform’s licensing change will fracture the market and the next thing will emerge.
I don't see any reasoning besides "I think it's not really needed".
The reasoning in the original article is shallow at best, and the presented alternatives are not discussed in the depth I was expecting for an article questioning the whole architectural basis of a very popular IaC stack.
On the second article, I don't see a self critique of the actual points raised during the original article.
The two articles sound much like a collection of statements based on personal opinion.
I see the value in an article like this for starting a discussion but not into taking any sort of conclusion.