Using Terraform Workspace for AWS multi account architectures
github.com
github.com
What we recommend people do is use modules to encapsulate the shape of your infrastructure and parameterize it. Then have each environment be a directory which instantiates the module (or modules).
This is more robust for a few reasons:
1. In most cases, as you scale, differences between environments will grow, with this approach you don't have to make a single root module act like a bunch of root modules via variables and conditionals, instead each environment is its own root module and if you need to do something unique in a particular environment, you can just implement that in the appropriate root module.
2. It's easier to see what environments are under management by inspecting the layout of the repository. With workspaces, you need to understand how whatever tooling you are using is executed because that is where the environments will be expressed.
Last weekend I also implemented what I call "Terralith" which is a proof-of-concept for how to treat a single root module as multiple environments in a principled way. I wrote a blog about the experience if anyone is interested: https://pid1.dev/posts/terralith/
> In particular, organizations commonly want to create a strong separation between multiple deployments of the same infrastructure serving different development stages or different internal teams. In this case, the backend for each deployment often has different credentials and access controls. CLI workspaces within a working directory use the same backend, so they are not a suitable isolation mechanism for this scenario.
For a practical scenario, you will often need different environments to roll out changes at different times, or to have other slight variance. If you rely solely on variables to be the only difference between environments, you will need a lot of tricky shenanigans to say, create a new dynamodb for a proof of concept only in “dev” but not in prod. Sure, you can use `count = var.env == “dev” ? 1 : 0` but this gets old fast.
Much better to make modules for common stuff, and then compose them in your different environments. Depending on the complexity of your needs, keeping good organization & practice around using modules can be a bit challenging, but it will definitely scale through composition.
Modules also important to make multiple copies of things within an environment, for example to have a cluster in us-west-2 and a cluster in eu-central-1, both are in production environment. I would assume if I started with workspaces I would rapidly hit a point where I want to use it as a module and then need to re-organize things. If you chose workspaces as soon as you want a second region you need a big refactoring, but if you are using modules, you just instantiate the module a second time in your second region.
- 1 Terraform workspace per environment (dev, test, prod, etc.).
- Managing changes to workspaces / environments is done with whatever approach you use for everything else (releasing master using some kind of CICD pipeline or release branches). The Terraform is preferably in the same git repositories as your code but can be separate.
- The CICD tool injects an environment variable into the build to select the appropriate workspace and somehow supplies credentials granting access to a role that can be assumed in the appropriate account.
- A region module / folder that defines the resources you want in each region. This is your "main" module that specifies everything you want.
- Minimal top level terraform that instantiates multiple AWS providers (one for each region) and uses them to create region modules. Any cross region or global resources are also defined here.
- The region module uses submodules to create the actual resources (RDS, VPCs, etc.) as needed.
This approach assumes you want to deploy to all your regions in one go. That may not be the case.
The problem with manually composing your environments is that you can lose parity across them. Over time it can be hard to point to what is the source of truth for your architecture. Like, this environment invokes these modules, but this other environment does not! Which is the correct one? But when systems cross team boundaries then they will diverge by design, using workspaces for such cases may not be a good idea.
You can have a directory per environment and a directory of shared resources that are used by all environments.
It seems like workspaces add a new construct to be learned and another thing to add to all commands without much benefit. Could we just stick with the simple way of doing this?
I agree it isn't a simplest way to do that, but I don't think that it add as much complexity this. Perhaps it could be more laborious for the point of view of architecture, but it could be easier to handle and maintain.
It's a pretty classic separation of code and config. Might not be intuitive to everyone I guess, but that separation is very beneficial I find.
For instance, adding a new environment is relatively trivial. Not something you do all the time granted, but I have had the need on occasions.
Same goes for removing an environment.
Things like terragrunt can be really helpful. I don’t use terraform without it.
Shoutout to the Waze team for creating it!
https://github.com/GoogleCloudPlatform/terraformer
We built out a large serverless stack on AWS, and we got a request from higher ups to convert it all into Terraform modules for portability and transparency purposes.
The Terraformer tool pulled in the entire stack and spit out the whole thing into tf files in less than 30 seconds.
Everyone was super impressed on the team lol.
This would only work if all developers on a team have synchronised the same AWS CLI config (which to me is like asking people to synchronise dotfiles, not something I'd be willing to do).
My go-to architecture for multi-environment tends to be this, as it lends itself relatively well to Git Flow (or GitHub Flow): https://github.com/antonbabenko/terraform-best-practices/tre...
We use a CLI command `notion aws-sso-login` that logs us into our main user account and adds a profile per delegated app-environment pair account you can access (like app-dev/collections-infra, app-prod/collections-infra) to the AWS CLI config file. This ensures at least the standard list of profiles is present on everyone’s machine whenever they have valid credentials. I have yet to hear anyone complain about this config file meddling.
Then in our terraform directories, we use direnv to set the AWS_PROFILE environment variable to the appropriate profile to manage that stack. You can always override if you need to use a different profile for some reason.