Two years with CloudFormation: lessons learned
sanderknape.com
sanderknape.com
My recommendation is to treat CF as a single point of failure. Once it gets in a broken state, you may have to destroy your stack and rebuild it. Even if it is fixable on paper, being able to just nuke a stack and replace it is a very good thing. This has happened to us multiple times and having a plan helps.
So what I do with elasticsearch for example is use 3 CF stacks (one for each AZ). This allows me to do things like rolling restarts in a sane way without having to do some flaky deep integration into CF to make it orchestrate a rolling restart without destroying my cluster state simply by replacing the stacks one by one.
If I were to build this again, I'd probably use terraform. Also, I'm looking forward to moving most of our stuff to kubernetes.
One of the more common scenarios where CF gets into a broken state is:
1) create new S3 bucket + something else (e.g. some Elastic Beanstalk env update)
2) something else fails, causing rollback
3) S3 bucket already contained data (e.g. your failed Elastic Beanstalk env update caused it to write data)
4) CF refuses to destroy the S3 bucket, entering a "rollback failed" state
In this cause, manually wiping the S3 bucket works well enough. But generally, it appears that CF works kind of when the updates you're making are really small, incremental updates.
Sometimes it gets totally corrupted and you need to nuke stuff, per your advice. This automatically leads me to the following suggestion: leave mission-critical data out of CloudFormation. Specifically, stuff like RDS databases which you absolutely never ever want to have destroyed: just provide the endpoint as an input to your CF template.
Check out the docs here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...
This a major pitfall when using DeletionPolicy=Retain with named resources. It breaks seamless rollbacks/rollforwards. If you rollback, in order to deploy again you need to either delete the named resources with DeletionPolicy=retain that were rolled back, or update your template to rename them all. It is such a huge pain.
A nice feature would be a "ForceDelete" deletionpolicy where it would delete the objects. You can even set this initially when creating a stack, and change it to "Retain" later when the stack is stable.
Totally agree btw that it's a huge pain initially, though once you know it it's also not that hard to work around.
There are ways around it as stated below, but I agree completely. I don’t bother with CloudFormation with cross shared, building block type infrastructure like RDS and ElasticSearch. It’s just not that much of a pain to spin up a database on each account. Besides, the characteristics of the databases in different environments are going to be so different, that you are going to either have parameters or FindInMap functions anyway so for all intents and purposes, you’re not running the same template anyway.
As the article said, changing any resource that’s exported from a CF template is such a pain, it would be better just to use parameter store if you can get away with it.
Try to avoid naming resources ("Name" property), so there would be no clashes. Use Ref (same stack) or ImportValue (other stack) to reference created resources. If you want, you may concatenate (Fn:Join "AWS:StackName", "-public-elb"). Cloud way is you replace things, rather than keeping it. It is convenient to know, that if I delete CF stack, everything is cleaned.
Speaking of S3, its better not to include bucket resource in CF until you know what you're doing.
> stuff like RDS databases which you absolutely never ever want to have destroyed
That begs for a separate CF stack, with template creating RDS and related things only, then export endpoint in Outputs.
The problem introduced by that approach is how to manage a large number of CF stacks. First I used a homegrown Python library to manage them, then switched to having Terraform manage CF.
At first Terraform on CF was just intended to be an expedient measure to facilitate migrating everything to Terraform, and eventually we did migrate to pure Terraform. But then we started hitting all the rough edges in Terraform. In hindsight, the hybrid approach had actually been more stable and manageable than using either tool in isolation.
Terraform also lacks a lot of the extra "smarts" that CF has; like rolling updates of any kind and some other higher level automation across different services. They take very much a blue/green approach which is beyond limiting for some services.
Ran into a few broken states using Cloud Formation via Serverless project. Luckily had been in the habbit of keeping "statefull" AWS resources such as queues, databases, and other stuff in their own stacks(or Terraform) to keep their stacks complication to a minimum and mitigate the impact of the more complicated stacks needing to be deleted..
Ultimately I started going with and promote a hybrid approach where Terraform makes sense; Terraform for base infrastructure(including most stateful resources) and CF for stuff like autoscale groups and etc.
- Good CF template is 10x less code for the same solution.
- No corrupted state problems.
- Native tool, supporting all properties of resources
Writing good CF templates takes good AWS knowledge, and system thinking, you group resources that belong together, it actually teaches you good architecting.
- Much better than Cloudformation at telling you what it's going to change before you apply the changes and the ability to record those changes. (much better than those dreaded 'conditional' changes)
- The ability to import changes if you found some that were done outside of Terraform. It's not perfect, or easy, but mostly doable.
- The ability to look at the code, the state file and the plan to get a good representation of what's actually deployed.
Those three are more significant than it looks, but together it makes sure you:
- Don't get into a situation where automation is broken and you can only recover by rebuilding the stack.
- Don't get unexpected downtime because a change replaces a resource unexpectedly.
- Being able to track, record and manage changes in easy to read diffs and plans.
CF: to look at template code only, to know what was deployed.
The latest example I have is the cloud formation used to generate EKS clusters(https://amazon-eks.s3-us-west-2.amazonaws.com/1.10.3/2018-07...) it clocks in at 168 LOC and the equivalent terraform (https://raw.githubusercontent.com/terraform-providers/terraf...) is only 53 LOC
resource "aws_s3_bucket" "x" {}Furthermore, the other comments on this post should disabuse you of the notion that there are "no corrupted state" problems with cloud formation: they happen all the time.
Disclosure: I worked on Terraform for quite some time at HashiCorp and am still a (community) maintainer of the AWS provider.
CFN has its warts, but I full-stop don't trust HashiCorp's operations or their attempt at a SDLC and I wouldn't trust my business's health to them as a company (and if the results of using it weren't enough, their clownshoes sales team's bad attempts to upsell would cinch it).
I mean, it's weird, because I agree with your statement that "The most important part of infrastructure engineering is being able to debug and fix things quickly by isolating issues to the smallest possible domain." And that's why I so strongly prefer Terraform, because I actually have control over the state file, how Terraform interacts with it, and I can move things in and out, and change things in-situ if necessary.
That. It's most dangerous thing for long term projects - third party tools and services. You want to minimise that to absolutely necessary ones.
As for state, I'm not sure what you did to corrupt your state, but we use Terraform to manage thousands of resources across dozens of AWS accounts for the past three years and haven't had any state corruption, except when a human messes up editing a state file by hand. Obviously in that case you back things up first (or hopefully you are using remote state with some kind of versioning). But the fact that you are _able_ to manipulate the state with the CLI tool, or by hand in extreme cases, is itself a huge advantage over CloudFormation, which has no such capability.
As for coverage, my experience has been that Terraform often has coverage of new resource types and properties _before_ CloudFormation. And it's extremely rare for any new features to take very long to show up in the AWS provider. Anything significant is usually picked up in 2-3 weeks from the API release at most.
That said, my experience has been that both CloudFormation and Terraform are irritating, just in different ways; they both are warty.
I do ultimately prefer Terraform - even in a single-cloud setup.
Recently, I decided to use Terrform over CloudFormation specifically because you can't create an EKS cluster (with nodes) in a single stack.
The rule of thumb that you should generally stick to CloudFormation if you are full bore invested into AWS has some truth.
My issues with CloudFormation are lack of control over rollbacks, missing features for existing and mature services like the above, and forcing me to use custom resources to do anything that vaguely resembles coding that Terraform does just fine like IP address math functions.
Unfortunately, this is far from the case. I can name 3 things off the top of my head not supported in CloudFormation.
1. The ability to create Route 53 records for Certificate Manager
2. Create a full EKS Cluster (with nodes).
3. EC2-Fleet.
* Error messages are overly verbose yet cryptic, and sometimes even unrelated to the actual error raised by the cloud provider themselves. Coupled with the lack of line numbering or other helpful identifier aside the unnecessarily long module hierarchy and debugging those scripts is a massive exercise in frustration and usually far more time wasted than really should ever be necessary.
* HCL is a hateful "language". The fact you cannot order stuff procedurally means you're constantly running into dependency issues on larger deployments. And dont even get me started on the "count" kludge to work around a lack of proper iteration.
* There is a lack of internal consistency with the support of different methods. Eg "count" does t always work with all resource types. Some resources cannot have properties defined with variables.
* Using calling modules requires so much bootstrapping code. It's just painful.
I get Terraform is the best we have for multi-provider deployments but their idea to create a superset of JSON only to then compile that back down to JSON anyway was such a poor decision in my personal opinion. I get the point was to have something that was accessable to non-programmers while still expressive enough for developers to use; however instead what they've created is a monstrous language that is too complex for the former group and too irrational for the latter.
I've been been very tempted to write my own Terraform alternative based on my experiences using it (and CloudFormation) - I even already have another programming language that Ive written a parser for and would be well suited for this type of application. But my time is pretty limited at the moment so I struggle on Terraform.
{
"foo": { "bar": 1 }
}
That said, you might be interested in Terraform 0.12 [0], which will be using some new HCL v2 that actually has first-class expressions and dynamic blocks (for loops). And, finally, a ternary operator that short-circuits. Unfortunately, the dynamic block stuff looks like it's based around for-loops and doesn't support just regular if-statements... but we'll see where that goes.
I wasn't aware about the JSON subset / Consul problem though. That's really interesting to read. It's funny because back when I was building test Consul cluster I did wonder why JSON was used for config instead of HCL. I guess now I know why.
Otherwise, about like you, I'm tempted to write a Terraform frontend that interfaces with existing providers...
Infrastructure as code tooling is all very primitive compared to what we take for granted writing most other traditional software but it will take some time and maybe another generation to do it well.
I know Terraform is the best tool we currently have (I even explicitly stated that I'm my previous post) but that doesnt mean there isnt still a massive room for improvement. Starting with the depreciation of HCL, in my personal opinion.
Plus the pace of development in nodejs worries me. All too often I've ran into issues where modules have changed and broken things downstream. When you're running infrastructure as code you really want to be damn sure your tooling is going to be consistent for years to come and I really don't have that faith in nodejs. Sure, if your a JavaScript developer you can manage it easily enough, but if you're DevOps who rarely touches JS then you really want your tools to be low maintenance. So to that end I wouldn't consider any nodejs projects for any serious production work given the kind of customers I work for (high availability stuff for some major brands). I might be fine but it's just not worth the risk.
#backend/state uses less priv keys
terraform {
backend "s3" {
bucket = "mystatebucketisunique-tf"
key = "states/tf/tf.tfstate"
region = "us-east-1"
lock_table = "lock-tf"
profile = "aws_tf_s3-prof"
}
}
Just, no.
[1]: https://github.com/awslabs/serverless-application-model/issu...
[2]: https://github.com/awslabs/serverless-application-model/issu...
Using exported resources and avoiding nested stacks makes it impossible to reach these numbers.
Never managed to get even close to that number, doing CF for 5 years :)
In my case, it's Lambda + API Gateway that's the main culprit. For each endpoint in an API, there's a resource for the lambda itself, and a managed policy, role, log group, subscription filter, API resource, method, lambda permission, model, and additional OPTIONS method for CORS purposes. With a setup like that, you can hit the limit with a moderately-sized API.
The biggest gripe I have with CF is that it's impossible to introduce existing components into a CloudFormation stack, so any legacy infrastructure has to remain manually managed.
It looks like internal products need to work with cloud formation to enable support, and aws doesn't have a consistent model here. It seems that they are fine with some products cutting corners and not offering support (like DNS based certificate validation)
Inconsistency within aws isn't all that surprising.
That said, one irritating omission I've had to deal with is not being able to add email subscriptions to SNS topics. The underlying AWS API is a bit odd - I don't think it provides an ARN until the subscription is confirmed.
The only reason I advocate for doing it is if a team will have a small infrastructure complexity ( like a basic ELB -> ASG -> RDS/EC2/S3 ) and they don't want to bring in more complex tools. Using ansible means you can use one tool to manage both your AMI's for immutable infrastructure and the infrastructure its self ( and can easily script your continuous deployment ). Once you start to really get a complex footprint getting a dedicated tool for infrastructure makes a lot of sense.
You may use Ansible for top level orchestration, deploying CF stacks and providing CF parameters. Just leave all resource creation to CF itself.
When I used CF a few years back (when it started) it was a pain (for those things it actually supported). I'm now using azure and ARM's integration with azure's cloud seems better to me.