Terraform 0.15 General Availability

What I want: Use Terraform programmatically, i.e. call "cdktf deploy" or similar FROM node or python and give users some scripts they can use where I can abstract away some of the difficulties of learning to use Terraform natively for simple use cases (i.e. deploy an S3-based frontend host). Ideally, I had intended to distribute some npm-installable packages which would run this stuff.

Out of curiosity what is it that people generally don't like about HCL?

We have are actually working on : "How to manage Terraform resources". We ended up having a conflict dev <-> ops where dev teams are messaging the terraform guys to create resources. For example, to create a database, it will take only 1 hour writing HCL, but 2 weeks of emails to align on the specs.

We are currently building something on top, to have resources that can be created in a self-service mode by the devs themselves. (Behind the scenes, it uses Terraform modules to generate resources that will comply with the company policies).

For the moment, it's a bunch of Jenkins pipelines. Having a CDK can actually help us a lot there. (Can plug it to a CMDB database, have a UI on top, etc)

Some core components that most will include as remote state, and you can discover and use as remote state stuff from other teams, too.

Quite basic, but something tells me this could very well be intentional, so that the enterprise offering has a clearly increased value.

This way, the per account setup (represented as a repo) is relatively small, common patterns are standardized, and there is still room for experimentation.

You can setup a workspace per subfolder, setup to automatic planning or not at each new commit, and put a team on it.

Main features I see : store the state and take care about the durability and the encryption + can create teams + cicd pipeline + manage variables.

There is a free tier until 5 active users also, good for experimenting it / showcase the advantages to stakeholders

- /modules holds a bunch of common modules. As an example, we have an aws_application module to setup application configuration in system manager, and an ecr repository for the docker image.

And then we our workspace folder (does not actually use terraform’s concept if workspaces). This goes down in specificity and uses terraform_remote_state for values from the previous workspaces. Our CI runs terraform automatically in this order:

- /workspaces/aws sets up our main aws account. No applications actually run here, it’s just iam setup and some random configuration values.

- /workspaces/production sets up a lot of the backend. We aren’t a big company so we don’t have to deal with cross region databases or redis clusters. This is also where we would call the aws_application module and setup a ecr repo.

- /workspaces/production-us-east-2 is where we setup the ecs cluster, task definitions, load balancers, and dns routes. We’re small, so we only have one region, but the idea is we could copy this folder to another region and horizontally scale super easy.

Then we have the same folders for our staging setup, only with some scale values and configuration tweaked.

Overall I’m pretty happy with this solution. It keeps any issues from spreading far due to the different state files. Though this can get pretty ugly if you need to use some region information in the environment workspace (production-us-east-2 values in production). I also can’t comment on how well it scales out past what we have done so far.

I can't say that this is the "right" way to do it, but it scales OK for our org (100+ engineers, 200+ services per environment, many third party and in-house providers). A single team owns the codebase, though all backend engineers expected to write and maintain their own infra code. Some highlights:

  * Monorepo with ~ 700 terraform modules, > 1000 terraform workspaces
  * CI/CD tooling to work out the graph of workspaces that need to be executed for a given change using merkle tree/fingerprinting, and build dynamic plan/apply pipelines for a given PR
  * Strict requirements about master being up to date, and serialised merges managed by a bot, with continuous deployment (i.e apply) on merge
  * Templated code/PR generation for common tasks
  * Tooling for state moves, lock management etc
  * Daily full application of all workspaces with alerting on state drift (e.g externally updated resources)

A few tips:

  * Find the right level of abstraction for breaking larger workspaces down into smaller ones. This should be determined by things like rate of change, security requirements, team ownership, and in some cases whether you have a flaky provider that you want to isolate. Size of state should also be a consideration - if a workspace takes 2 minutes to plan, it's too big
  * If you start to use lots of remote states, wrap all remote states into a module with a sane interface to make consumption easier. You can also embed rules in this like "workspace X cannot consume from state Y" (e.g because of circular dependencies or security considerations)
  * Never embed a provider within a module (I think this is enforced in newer TF versions)
  * Terraform is a hammer that will can tackle most nails, but for several problems there are more appropriate tools

Personally I have a nix shell file pinned to the exact version of terraform (as in, commit hash on the nix-packages repo) we use in every repo and just switch to that shell before doing anything.

Terraform 0.11 to 0.12 is by far the most difficult of the versions to upgrade between. I am really sorry about that. The other upgrades should be relatively minor as long as you read and follow the upgrade guides and upgrade one minor version at a time (0.11 => .12 => .13 etc.). There are rough edges for very specific cases but most of our customers were able to upgrade from 0.12 to subsequent versions same day without issue.

Breaking changes and difficult upgrades is not something we want to do with Terraform (0.12 being a big exception as that was a very core "reset" so to speak). The reason there have been these changes in these recent releases is that we've been ensuring Terraform is in a place for 1.0 that we don't have to have difficult upgrades.

You can see this path happening in each release:

- Terraform 0.15: state file format stability

- Terraform 0.14: provider dependency lock file

- Terraform 0.13: required provider block (for more deterministic providers)

- Terraform 0.12: stable JSON formats for plans/config parsing/etc. (particularly useful for interop with things like Terraform Cloud and 3rd party tooling)

This was all done to lead up to a stable 1.0 release.

As noted in the blog post, 0.15 is effectively a 1.0 pre-release so if everything goes well, we've MADE IT. For 1.0, we'll be outlining a number of very detailed compatibility promises which will make upgrades much easier going forward. :)

This approach worked for me in various setups from a small SaaS company to a major travel company as well as personal projects without issues.

Looks like I need to clear my schedule in a upcoming sprint to get this done so the pain doesn't get even worse :)

0.12 -> 0.13: Well, we had to add what feels like a million required-provider-blocks and then some more. And sometimes it was tricky to pinpoint the module pulling in a default provider and crashing - `terraform providers` and `terraform graph` help there. The graph is easy to grep through to find the resources and modules pulling in wrong providers. And in the beginning the error message we got when we had to run 'terraform state upgrade-providers' was .. obscure. In newer versions, that message is much better.

0.13 -> 0.14 just happened and now the lock files are slowly piling in on demand.

  - Don't upgrade directly from 12 to 14, go to 13 first
  - If you have warnings after moving from 11, fix them first
  - Run the 0.13upgrade command in your code that will generate the required_providers
  - Run terraform-v13 init
  - Change to correct workspace if using some
  - Run terraform-v13 plan which will probably fail due to the new explicit required-providers rule, if that happens you need to modify the state with the correct providers https://www.terraform.io/upgrade-guides/0-13.html#why-do-i-see-provider-during-init- . In my case I have a lot of modules so I created an script that automated that process
  - Execute again terraform-v13 plan and verify that it will not make uncommon changes
  - Then run terraform-v13 apply.

The point of using TF for many is reproducible infra and change approval workflows. So avoiding manual changes via the web consoles is what people are striving for.

>manually fix

You can if you need to. TF can ignore changes to certain aspects of resources like desired count of ECS tasks which might autoscale up and down and you can always make the change manually and then update the code to reflect the change. That's a bit of a no-no though. Like pushing code without a review.

For example, at my work this is what i do to apply changes to our AWS setup:

1. Fetch the latest version of our git repo.

2. Create a new git branch named after the Jira ticket im working on.

3. Solve the jira ticket by modifying the terraform code accordingly.

4. Submit a pull request and assign one of my colleagues as reviewer.

5. They review my solution, tell me to correct some issues that there might be, or straight up approves my solution.

6. My PR is merged into master.

7. I download the latest master version and apply the codebase.

This way we always have at least two people verify any changes to our infrastructure, minimizing the risk of fuckups and ensures solutions are as good as possible.

I think tf really shines when you start using multiple providers to manage things outside of the cloud though. In my last gig I was for example using the workspaces feature and auth0 provider to have separate auth stacks for our different envs, being able to use values created by one providers resources in another's was nifty

This is a feature, not a bug. Terraform is a tool to (reproducibly) enshrine your infrastructure in code. 'Minor manual fixes' are often left undocumented and known only to 1-2 people, and suddenly become major problems when your infrastructure comes crashing down and nobody knows why it used to work.

The naive approach that tries to do this without state in code goes something like:

a) invoke the remote API to look for something that should exist b) if it doesn't exist, invoke the remote API to create it c) use the property of the thing to do something else

What Terraform does is cleanly separate your state (don't look stuff up you already know exists) from the API (which is handled by an open-source library, aka the provider) from your configuration (which can now be declarative). Because the state and API calls are abstracted away, the configuration itself is much more clean, easy to reason about, and easier to maintain.

That's why I like Terraform.

Edit: "Also, I created a thing 2 years ago and now I need another one. Oh, here it is in Terraform." It's incredibly nice not to have to re-learn how to make that thing, and remember what its quirks and dependencies are.

In short, I consider Terraform exactly like I consider shell scripts: it's not perfect, not by a long shot, but once you have it working it tends to stay working and you don't have to invent it again the next time you need to do something.

For me, the biggest thing is, when I go into AWS I struggle to find everything that is intrinsically linked to another resource. Say you have a lambda, to find which iam is linked to it, and what permissions it has is 2 separate tabs, then another for e.g. security groups, probably more tabs for other things. While using aws-cli makes it slightly easier, it's still a lot of effort to do this effectively.

With terraform I can look in one repo that has all the above, often in the same file too. Finding out what your infra looks like is a lot easier.

Regarding the state, you should not be touching your infra outside your code, if you do (e.g. while you're testing in dev), you should make the same changes in tf once you've confirmed it's what you want, and otherwise you undo those changes.

With further automation (e.g. tfcloud) you can even enforce these things by auto applying workspaces which ensures manual changes are always undone.

`terraform refresh` does this.

(Disclaimer: I am an ex-core-maintainer of Terraform)

After that perhaps you vowed to take better notes, so the next time you do it that way, but then 6 months later you find that you missed some detail, or there was some changes in between that were not recorded.

So then you decide you're going to do everything by API and save the commands, so you write a bunch of bash scripts that execute against the AWS CLI. Over time you add several more for different operations (eg. adding an instance, configuring new ssl cert, etc), but the list of scripts grows very long, and you find that each one makes a lot of implicit assumptions about the state of the infra when its run, so you end up with varying degrees of confidence in the scripts depending on how often you run them.

Now you are primed for Terraform. At this point you realize the hard part about cloud configuration is state management. Furthermore, you realize there are some common patterns of how different components and APIs interact with regard to serial dependencies and operation idempotency, however the specifics vary by service and by use case. Terraform gives you a standard substrate for state management, and a framework for developing service "providers" that know how to interact with APIs and map them to state. All of this happens in code which is declarative, can be code reviewed, and state which can be centrally tracked and shared among a large team for a clear audit trail.

There's definitely a learning curve, but once you learn it the overhead is pretty small compared to the benefits, even for small teams IMHO.

In my opinion if you're doing manual fixes you're doing it wrong. Let's say you do your manual fix in your Dev environment. Do you remember to do it in Prod/whatever other environments you have? Are you sure you did the EXACT same thing? Did you change 5 other things trying to fix it first?

You end up with so many different deployment environments that are unique 'snowflakes', and when something breaks in one it might not affect the others cause they're in totally different states.

It's a nightmare.

In my opinion, infrastructure as code is the only way to do it in a serious environment.

I find this to be the best advantage to Terraform. These small changes make your infrastructure harder to re-produce in the future; and unless you have strong documentation rules, you lost what is there in your infrastructure.

I had an old boss who fixed minor mistakes in stored procedures on the live server and then wondered why they broke again when builds went out...

Like WTF? Do they think I have better chance reaching them than them? Tried then TF and it was much simpler and everything worked as expected.

On AWS though, seriously, TF is inferior.

It would be nice, on the first run of terraform init, if it would check for the remote state bucket and then ask if you would like to create it if it does not exist.

That said, I've worked on/with a team enabling some fairly advanced & streamlined capabilities across all three clouds because of Terraform. The abstraction matters, and ability to model scaled platform components and architectures as singular units is important. The open source community is great.

Not applicable if you're deploying infra across cloud (we do that across a dozen cloud providers!). Without terraform, it'll bring us tears.

> Or about when it just loses tracks of resources.

Never had this in years of using terraform. These should be qualified with how often they occur.

> Terraform does not automatically rollback in the face of errors. Instead, your Terraform state file has been partially updated with any resources that successfully completed. Please address the error above and apply again to incrementally change your infrastructure.

Checkout the code from before the change and apply again. I've done this numerous times and it's been as good as a natively supported rollback.

> HCL? A disaster.

Why is it a disaster? I find it predictable, ergonomic and easy to write. I also really like the JSON compatibility. But this one is I guess mostly taste.

> it's a terrible tool

I've found Cloudformation to be an abomination as far as user experience goes! Every time I used it, I lost a bit of my soul.

More often than not with Terraform, another apply will fix the problems you are complaining about. But if it doesn’t at least you have the power to fix it unlike with CloudFormation.

But ultimately whatever tools you use you will have trouble if you are unwilling to engage with them and learn how they work.

Business critical readiness is not measured whether something has or hasn’t got a “1.0” tag on GitHub.