Hacker News Reader

Comment by qaq | Hacker News Reader

A bit tangential but why is CloudFormation so slowww?

The reason by boss tends to give is that it’s made by AWS, so it cannot possibly be bad. Also, it’s free. Which is never given as anything more than a tangentially related reason, but…

It… definitely isn’t free. Have you ever looked at the “Config” category of your AWS bill?

I figure it's because AWS can get away with it.

AWS deploys using cfn internally

This is only partly true.

The foundational services do not use cfn for the vast majority of their deployments.

This is just anecdotal but I have found anytime a network interface is involved, it can slow down the deployment. I had a case where I was deleting lambdas in a VPC, and connected to EFS, that the deployment was rather quick but it took ~20 minutes for cloudformation to cleanup and finish.

> A bit tangential but why is CloudFormation so slowww?

It's not that CloudFormation is slow. It's that the whole concept of infrastructure-as code-as-codd is slow by nature.

Each time you deploy a change to a state as a transaction, you need to assert preconditions and post-conditions at each step. If you have to roll out a set of changes that have any semblance of interdependence, you have no option other than to deploy each change as sequential steps. Each step requires many network calls to apply changes, go through auth, poll state, each one taking somewhere between 50-200ms. That quickly adds up.

If you deploy the same app on a different cloud provider with Terraform or Ansible, you get the same result. If you deploy the same changes manually you turn a few minutes into a day-long ordeal.

The biggest problem with IaC is that it is so high-level and does so much under the hood that some people have no idea what changes they are actually applying or what they are doing. Then they complain it takes so long.

Thing is Terraform is faster

50-200ms per poll is one thing, but realistically we’re talking 30+ seconds for the smallest of changes even on new resources. Why does it take so long to spin up an ec2 instance (when fargate can do it in seconds assuming you’re not rate limited by the API) or lambda can do it also in milliseconds. Those machines are already running, why does it take 3 minutes to deploy Ubuntu or Debian from a blessed AMI?

Fargate is running containers, Lambda functions. They use Firecracker microVM while EC2 uses full VM. EC2 instances does lot more setup, using bigger image, and user setup. My guess is Firecracker is designed for smaller VMs and can’t support EC2 features that people need.

> It's that the whole concept of infrastructure-as code-as-codd is slow by nature.

> If you deploy the same app on a different cloud provider with Terraform or Ansible, you get the same result.

Nope, Terraform is way faster. Anyone who has switched between them on the same project can attest to this.

Also, Terraform does not get into “UPGRADE_ROLLBACK_FAILED”-style unrecoverable states nearly as easily. This happens to me all the time with Cloudformation/CDK. So my second question after “Why is Cloudformation so slow?” would be “Why is Cloudformation more error-prone when it’s also slower?”

It very much depends on the project. TF has all sorts of slowness and failure modes all its own.

FWIW, my approach to IaC has been to focus on the “I” with CloudFormation — the networking, storage, IAM, other AWS primitives and etc. This stuff doesn’t change as often, and safe/reliable deployments are more valuable than quick ones.

The behavioral parts (aka. application, stuff running in a VM of some kind or something declarative like EventBridge rules or StepFunctions) I keep separate and prioritize quick turns. CodeDeploy can, for example, update code on EC2s in single-digit seconds.

I’m building systems that are a little more integrated in AWS than most folks, perhaps, which makes this approach a good fit. I do dozens of deployments a day (not an exaggeration — 21 so far today on a light day), including a couple infrastructure updates.

I think the secret here is not buying into meme-like simplifications and instead deliberately design an approach that works for your goals.

Reply on news.ycombinator.com