Hacker News Reader

Comment by merlincorey | Hacker News Reader

I am in agreement with everything generally except for CloudFormation and setting up Route53 zones manually.

I use Terraform to setup my infrastructure as service on AWS, including Route53 zones.

After the latest 0.12 upgrade to the language, Terraform is quite a bit more user friendly than CloudFormation, and importantly, not locked down to just Amazon -- it supports multiple clouds and on-premises solutions for declarative orchestration of resources.

View on HN

My company[1] has written 100's of 1000's of lines of Terraform code as part of a commercially maintained library of prod-grade Terraform modules. We've also used Terraform to setup 100+ teams on AWS with prod-grade infra, and I can confirm that Terraform works very well for robustly launching on AWS.

There's also a fast-growing ecosystem around Terraform: lots of open source modules, automated testing frameworks, and a growing number of tooling solutions. In the early years of Terraform, bugs and stability were major issues. With 0.12, the maturity factor is becoming very compelling.

On a separate note, I'm surprised the author endorses plain old EC2 over Docker. I get the point of "choose boring tech," but it seems like launching your app on EC2 requires a whole bunch of rework around automation that's already done for you in ECS, EKS, or dare I say even Elastic Beanstalk + Docker.

[1] https://gruntwork.io

> I get the point of "choose boring tech," but it seems like launching your app on EC2 requires a whole bunch of rework around automation that's already done for you in ECS, EKS, or dare I say even Elastic Beanstalk + Docker.

I have a service that has to run tasks immediately that can take a while to complete, often in a minute, but occasionally taking hours. (And we're working on the obvious solutions of making it both fast and interruptable, but this is tricky.)

With ECS, if you want the automation to place your services, you run into trouble with these when you're deploying a new version because ECS assumes that tasks can be stopped quickly.

Especially, if ECS has asked a task to stop and the task is taking time to shut down, it can't deploy new work to that host until that process reports that it's done.

Does EKS handle this better? I'm inclined to drop ECS entirely and just bake an AMI because the automation feels like it's getting in the way more than anything.

What are the downsides of using terraform? We are currently in the process of redoing a lot of our infrastructure and are considering Terraform. We had some bad experience in the past with AWS (probably 12-18months ago) and Terraform especially when it comes to manual changes to resources for environments where manual changes for testing purposes are common (think changing security group rules for example). It resulted in us having a broken state and being unable to apply changes to our Terraform deployment without tracing the manual changes and undoing them, so I'm a bit cautious about moving forward with terraform. Have you experienced this recently? I'm intrigued by your comment and would love if you could expand on it.

Ideally, don't allow manual changes to happen. It's not that hard to setup for different environments and testing, so IME, it's not been much of an issue.

However, if you really can't change your ways of working, which I understand if you can't, then try out the "terraform refresh" command. I've been importing state recently, to move some of our own infrastructure over to TF, and have found it to be quite useful for things like manual security group changes. Basically, I'm building things up bit by bit, and when one of my states gets out of sync I've been updating the local config and running that command, which brings the state back in line.

In general, once you get your workflows sorted out and running for a while, you're unlikely to have any major issues with Terraform. Just make sure to use remote states and version them whenever you can (for example, turn on versioning on the S3 bucket if you use S3 as the remote).

Terraform will generally just undo the manual changes for you. The issue is trying to mix manual and automation.

Author here. Fair. I never used Terraform beyond playing with it briefly, so I can’t really comment on the differences. I tend to choose CFN mostly because I’m familiar with it.

I'm in charge of cloud infrastructure at my company, and I've found that sticking with CloudFormation on AWS is more powerful and versatile than TerraForm, since TF isn't always a perfect mapping, where CFN is, with the caveat that it's sometimes months behind what their API can do.

To that end, over the years, I've come across a couple of tools which help write CFN templates and manage rollout keeping cross-stack dependencies in mind. If anyone does a lot of CFN, I highly recommend Troposphere (python abstraction for cloudformation) [1] and Stacker (manager stack rollout via dependency DAG) [2]

1: https://github.com/cloudtools/troposphere 2: https://github.com/cloudtools/stacker

Fwiw terraform also lags behind what the underlying apis can do -- but it's a community effort and you can contribute to add the functionality you need, whereas with CFN you can only ask your TAM (iff you have one!) to ask the internal team(s) (not sure if only CFN team or only the service team or both have to implement something) to prioritize your feature request.

As a counter point: my org users both ansible and cloud formation.

If you're use case is to create 10+ objects in parallel (as a response to a customer action) I really like cloud formation...

We built a system around it, and I love the robustness of CF... It's "fire and forget". We can literally create a virtually unlimited number of stacks in parallel and check on them later to make sure they are successful... Granted this is probably an unusual use case, but we user a stack to represent a customer... In essence... And we're providing a real (temporary) server or set if servers for student/education (so it's not like we can redesign this for something other than making EC2 instances). We can support an influx of 100 users per minute using CF, but using Terraform or ansible we had to spin up temporary worker nodes to run their scripts and could only contact as many stacks as we had worked nodes... So CF let's is have one $30/month server super nearly infinite scalability.. but ansible was requiring us to spin up nodes 1:1 with how many we wanted to construct in parallel and Terraform looked no better.

In addition.... And on this I can't speak for specifically Terraform.. but we also have been burned because community modules changing and breaking backward compatibility... We've been version locked on ansible because version 2.6 broke security groups and version 2.7 broke another module... These are the popular built in but community maintained modules... But people redesigning modules don't know all use cases that could be available in CF (like referencing a security group in another account across a VPC link).. so a rehearsal expression update and presto.. a valid SG reference is no longer usage in that ansible version.

Since we use Infrastructure as Code, obviously we still see there are still benefits. I 100% recommend codifying your system! However, I still like to throw in caution that these solutions are not typically as featured as the native tools. The third party tools are also not always the "correct" solution. In addition, "cloud agnostic" seems like a misleading term, since you can't pick up your Terraform or ansible code and go to another provider. So I think it's more accurate to say "they can do multi-cloud orchestration of components" than "they are cloud agnostic".

Yeah. Terraform is the mature standard for this. To me it feels like OP isn't that familiar with large scale infra work.

Author here. I helped manage thousands servers at Amazon across 20 regions. Obviously Terraform wasn’t an option :)

My company runs AWS cloud all 20 of the non government regions as well, google cloud, and our own data centers around the world. Yeah certain regions (hello eu-north-1) are problemmatic, but we have work arounds for that. Terraform has been fine.

> Obviously Terraform wasn’t an option :)

Why?

Because I worked at AWS for the last 8 years :)

Tbh, I don’t think it’s prohibited, but CloudFormation was the default choice.

You probably would have run into security issues with using terraform as each version would have to be vetted and any extensions you use

The Terraform AWS providers are maintained by AWS engineers, but the rest of the code will still need vetting.

Sometimes Terraform even supports new feature earlier than CF. It seems a separate team inside AWS to work on CF.

One of the nicest things about my current infrastructure is terraform + ECS + Route53 + ACM. There's basically no work required to set up a new server behind TLS loadbalancer and get the hostname registered, etc.

Check out CDK

Reply on news.ycombinator.com