Hacker News Reader

Comment by akurilin | Hacker News Reader

What are people's thoughts on Cloudformation vs Terraform for a project that only ever expects to use AWS?

CloudFormation covers all AWS features in full detail. OTOH, you end up reading and editing tons and tons of JSON that were never meant for human consumption. I don't know what goals Amazon had when they came up with CF, but if human readability was one of them, then they've failed that goal.

I've seen a lot of people start writing their own CF management systems, in Python or whatever, and end up with a lot of infrastructure defined in the logic of the homegrown management tool - like a badly written PHP page where presentation and logic are all mixed up. You see that once, you never want to go there again.

Terraform does not cover all AWS features, but the exceptions are few and tend to be lesser used features. The file format is human readable (this is subjective to some extent), and can be composed and modularized easily.

https://opencredo.com/terraform-infrastructure-design-patter...

https://github.com/hashicorp/best-practices

https://atlas.hashicorp.com/help/intro/use-cases/multiple-en...

https://www.terraform.io/intro/getting-started/modules.html

https://www.terraform.io/docs/state/remote/index.html

I'm not a big fan of buzzwords and soundbites, but Terraform comes pretty close to fulfilling the ideal of "software-defined infrastructure" in a way that is accessible, easy to use, easy to expand, and just makes sense - in the way a good programming language just makes sense.

Just a note, even Cloudformation is not covering all AWS features. It's more complete but not 100%.

Have you seen https://github.com/bazaarvoice/cloudformation-ruby-dsl and do you consider it an instance of this PHP-like stuff?

Terraform often beats CloudFormation to the punch for support of AWS services. Terraform is also far more friendly in terms of being able to preview what's going to happen and roll things back. If Cloudformation fails, it's really hard to fix, and too often you need to contact AWS to unstick a CF change. Basically you have no control over the process and you just have a poor web UI or a poorer API to interact with. When Terraform fails, the newest version gives you tools to edit and update your state and import existing resources, and even when things get ugly you are still in total control. In short, I have no reservations about endorsing Terraform over Cloudformation.

I just wanted to offer the other side. I do devops, we use Terraform, and we've had times when Terraform took action that was not present in the plan generated before applying, in production environments. We've also had times when it'll fail mid-apply, be stuck in an inconsistent state (IAM profiles and autoscaling group configurations, specifically), and require manual recovery through modifying or removing resources via the AWS console (although this might be better in the latest version).

We'll continue to use Terraform at my day job, but I don't recommend it to colleagues nor clients if they use only AWS resources. Simplicity is the ultimate sophistication, and adding yet another abstraction does not contribute towards that philosophy.

We consider these kinds of issues the highest priority and always work to resolve this ASAP. The resolutions very often include not only the fix but protections to ensure that unintended actions don't bypass anything in the future.

It's really unfortunate you had these experiences and I can't back up the statement when I say that this isn't normal and every release certainly does improve stability in a big way. Terraform is a very, very heavily tested (unit + acceptance) project and we fight very hard against regressions with multiple tests per bug in many cases at different levels.

I'd encourage you to continue giving future versions a shot and if you experience anything like this again, let us know and we'll react to that quickly relative to other types of issues.

Thanks for the reply Mitchell! Definitely don't want my criticism to come off as unconstructive. As I mentioned, we'll continue to use Terraform; I'm optimistic it'll improve as Hashicorp iterates. I'm just naturally conservative coming from an ops background.

I try to pick boring technologies that are battle tested through years of tech sector use; there's no glory for being on the bleeding edge, with only pain when things go south. So goes the ops struggle.

Yes to all this.

MIGBY, we have also been bitten in production in much the same way. If you are using only AWS resources there is no need to use terraform. Using it at all seems like a questionable decision to me since it's still firmly in "early adopters only" status.

If you're AWS-only then CloudFormation is the way to go. And IMO it will continue to be for a long time, until terraform is much more battle-tested and -hardened. Unless of course you're operating in an environment where you can afford to be exposed to the cutting edge of infrastructure-as-code. I definitely am not.

This matches my experiences, and my own recommendations. Multiple clients of mine have independently coined "terrafucked" for when Terraform frags its state. It's usually recoverable but it shouldn't under any circumstance happen, and regardless of my other beefs with Terraform (which, to be fair, aside from the quiet insanity of HCL, seem to be decreasing!) because of that risk profile alone I can't recommend it for AWS-only clients right now.

Personally, I use CloudFormation for my own stuff, because aside from reliability concerns, giving up on cfn-init to not write JSON (when I don't now, I use cfer, a Ruby DSL) is totally no bueno for me. Terraform offers no direct equivalent, though I've had HashiCorp sales people try to push Consul chewing-gum solutions that I'd have to manage much more directly than CFN metadata/cfn-init have been for me. Maybe in a couple years it'll be somewhere where I feel safer recommending it.

That's not true anymore. Cloudformation supports change sets that you can apply, similar to Terraform. They're not really human readable, but they're there.

They also allow you to manually get out of the UPDATE_ROLLBACK_FAILED state, which was usually where you needed to contact AWS.

Cloudformation also supports Lifecycle events. They are a bit complicated to set up right, but they allow you to tell CloudFormation how to wait and how to proceed in response to resource signals (for instance, the deployment can wait or roll back if new instances in an ASG fail to be deployed). Terraform has none of that. It has "create before destroy", and "depends on", but that's about it AFAIK. If CloudFormation fails it will at least attempt to roll back to a sane state (and usually succeeds). Terraform will give up and leave you to clean up the mess.

Apart from that, Terraform has an advantage because of the modularity, ability to easily bring resources in, reference other resources in remote states etc.

Terraform was built using CloudFormation as the inspiration, but development became stale quickly and, at the same time, CloudFormation introduced some unique features. Given Lambda and Custom Resources, you can manage foreign to AWS resources now, and CloudFormation does not run on a single machine, but on multiple ones, I will speculate, using Simple Workflow Service. So, I pick CloudFormation as a reliable serverless service; I generate the JSON using troposphere, and I'm more than happy - HCL, which pretty much nobody outside of HashiCorp uses, can't beat Python! I am also not sure how viable HashiCorp is as their business model is strange and the sudden death of Vagrant should be a lesson learned!

I'm fairly new to both, and I initially setup some infrastructure with CloudFormation. I found it difficult to use and the JSON impossible to write by hand. You need to use a tool like their online editor to generate it - at least initially. It was also tough to extract identifiers to variables that can be changed outside the templates.

I'm using Terraform now, and I like it much better. It has its own language for defining resources, but I find it very readable and easy to write by hand.

My goal is to have my "infrastructure as code" so it can be tested in a sandbox and changes can be reviewed as pull requests. Going forward Terraform is my choice.

There's a lot more to terraform than just infrastructure providers. You can use it to render templates, generate certificates, setup datadog monitoring, add users to a github organization, configure a Postgres database, ... and then use those things as part of your infrastructure definition.

Also, HCL is way more readable than JSON. You can actually do a code review on a terraform plan.

And terraform as a CLI tool is pretty easy to use and expand. On top of this, you can write your own providers fairly easily and it's a more 'standard' tool as in, you don't get vendor lock in.

On top of all this, terraform is an open source Go project: you can easily make your own tools and import it's internal packages to hack things to your needs (caveat: internal packages =P).

If you end up using CloudFormation, don't write the JSON directly. Your programming language of choice should have a wrapper that maps objects to CloudFormation JSON - do your thinking there, use all the features your language offers, and consider the JSON you dump out at the end as an artifact you deploy (which happens to be readable). For Python, this is https://github.com/cloudtools/troposphere - I can't vouch for how accurate other languages' versions are, but I've never had a case where Troposphere didn't generate the right JSON for what I asked it to do.

CloudFormation is also nice in that it has access to a few API features which Terraform doesn't - specifically wait-conditions, automatically rolling-back updates when there's a problem, and a lot of the magic around UpdatePolicy/rolling deploys (although that doesn't work the way one would expect - a story for another day).

Having said that, Troposphere is pleasantly readable and has nice docs. The cross-platform integrations (for example, CloudFlare) add to the "just works" feeling. Some things are a pain to do in Terraform due to its strictness around being declarative and not having conditionals (e.g. having production and development environments that are similar, but not exactly the same), but there are some well-known hacks around them.

It also takes a very strong position around failed updates: it'll stop mid-change, tell you something's broken, and have you fix it. In the same situation, CloudFormation would roll back the changes to the state you were in before you started. Which one of those two failure modes you prefer is up to you. There are also some issues around having passwords and the like in your statefiles when you use RDS. You can avoid this if you go whole-hog into the HashiCorp stack (TF + Vault + Consul + etc.), but it's bugged me.

Lastly, If you're going to build something big and complex in TF, I suggest you do some reading of Charity Majors' rants on TF (https://charity.wtf/tag/terraform/). She has probably already run into the problem you're going to have.

So which would I use? Depends. If I want to create the same environment in a bunch of places (which also includes blue/green deploys), TF. If I want to do more complex things than that, CloudFormation.

I stayed away Terraform because (1) I believe in templating CloudFormation template instead of relying on another abstraction built by someone, and (2) Vagrant was deprecated for the rise of Otto, which I thought was a disappointment and since then I have been very careful at choosing projects like these, and (3) I use Ansible for config management so I want to avoid another tool.

In the end, you choose your toolset and I have made a deliberate decision to stay as close to the provider's bare metal solution and build from there. I can build my own tiny "abstraction" a couple hundred lines of Python generating CF template and parsing the template into readable human format.

I found Terraform's config file syntax to be much simpler and understandable than Cloudformation's JSON. However, there are a couple edge-cases around some of AWS' newer services, such as Lambda, that didn't seem well-supported, as of a few months ago. If I was using Cloudformation, I probably would have written a tool 20% as complete as Terraform to generate the json from another config file, so I'm glad Terraform exists so that I didn't have to.

We ended up going with Terraform because it's much easier to handle more complicated use cases. When we were using Cloudformation stacks everything had to be done through hooking together JSON. I found that to do anything complex I needed to drop into a combination of Python with boto3 and Cloudformation anyway.

the introduction of cloudformation changesets pretty much make terraform strictly inferior. you get update policies, rollback on errors and stack outputs as a bonus

working with json can be a pain, but there's lots of great tools out there that output cfn templates from either dsls or nicer formats

I've used both. I like Terraform better as a pure CLI tool. If you like UI driven configuration, CF may be a better choice.

Even if your project only expects to use AWS can you say that for certain of all your projects?

Reply on news.ycombinator.com