Hacker News Reader

> My recommendation is to treat CF as a single point of failure. Once it gets in a broken state, you may have to destroy your stack and rebuild

One of the more common scenarios where CF gets into a broken state is:

1) create new S3 bucket + something else (e.g. some Elastic Beanstalk env update)

2) something else fails, causing rollback

3) S3 bucket already contained data (e.g. your failed Elastic Beanstalk env update caused it to write data)

4) CF refuses to destroy the S3 bucket, entering a "rollback failed" state

In this cause, manually wiping the S3 bucket works well enough. But generally, it appears that CF works kind of when the updates you're making are really small, incremental updates.

Sometimes it gets totally corrupted and you need to nuke stuff, per your advice. This automatically leads me to the following suggestion: leave mission-critical data out of CloudFormation. Specifically, stuff like RDS databases which you absolutely never ever want to have destroyed: just provide the endpoint as an input to your CF template.

You can set the DeletionPolicy attribute to "Retain" to work around this S3 issue. CloudFormation will successfully rollback without attempting to remove the S3 bucket. You can then do so manually yourself before trying to deploy it again.

Check out the docs here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...

You better do so before deploying again, because the roll forward will break since the resource already exists.

This a major pitfall when using DeletionPolicy=Retain with named resources. It breaks seamless rollbacks/rollforwards. If you rollback, in order to deploy again you need to either delete the named resources with DeletionPolicy=retain that were rolled back, or update your template to rename them all. It is such a huge pain.

True, but it beats the alternative where CloudFormation deletes objects that you didn't want deleted. The underlying issue is that the S3 objects are outside of the CloudFormation scope, thus it takes no risk and doesn't delete your objects.

A nice feature would be a "ForceDelete" deletionpolicy where it would delete the objects. You can even set this initially when creating a stack, and change it to "Retain" later when the stack is stable.

Totally agree btw that it's a huge pain initially, though once you know it it's also not that hard to work around.

My preferred behavior would be for CFN to not barf when rolling forward. In other words, to be able to assume control over a resource that already exists.

Specifically, stuff like RDS databases which you absolutely never ever want to have destroyed: just provide the endpoint as an input to your CF template.

There are ways around it as stated below, but I agree completely. I don’t bother with CloudFormation with cross shared, building block type infrastructure like RDS and ElasticSearch. It’s just not that much of a pain to spin up a database on each account. Besides, the characteristics of the databases in different environments are going to be so different, that you are going to either have parameters or FindInMap functions anyway so for all intents and purposes, you’re not running the same template anyway.

As the article said, changing any resource that’s exported from a CF template is such a pain, it would be better just to use parameter store if you can get away with it.

> DeletionPolicy=Retain with named resources

Try to avoid naming resources ("Name" property), so there would be no clashes. Use Ref (same stack) or ImportValue (other stack) to reference created resources. If you want, you may concatenate (Fn:Join "AWS:StackName", "-public-elb"). Cloud way is you replace things, rather than keeping it. It is convenient to know, that if I delete CF stack, everything is cleaned.

Speaking of S3, its better not to include bucket resource in CF until you know what you're doing.

> stuff like RDS databases which you absolutely never ever want to have destroyed

That begs for a separate CF stack, with template creating RDS and related things only, then export endpoint in Outputs.

I agree with splitting up CF stacks this way. It reduces the the blast radius, that saved me on a number of occasions when stack updates went sideways.

The problem introduced by that approach is how to manage a large number of CF stacks. First I used a homegrown Python library to manage them, then switched to having Terraform manage CF.

At first Terraform on CF was just intended to be an expedient measure to facilitate migrating everything to Terraform, and eventually we did migrate to pure Terraform. But then we started hitting all the rough edges in Terraform. In hindsight, the hybrid approach had actually been more stable and manageable than using either tool in isolation.

You should take a look at AWS Code Pipeline, which natively support creating/updating cloudformation stacks.

Same waiting with Terraform. If you aren't automating windows installs and/or imaging, count yourself lucky :)

Terraform also lacks a lot of the extra "smarts" that CF has; like rolling updates of any kind and some other higher level automation across different services. They take very much a blue/green approach which is beyond limiting for some services.

Ran into a few broken states using Cloud Formation via Serverless project. Luckily had been in the habbit of keeping "statefull" AWS resources such as queues, databases, and other stuff in their own stacks(or Terraform) to keep their stacks complication to a minimum and mitigate the impact of the more complicated stacks needing to be deleted..

Ultimately I started going with and promote a hybrid approach where Terraform makes sense; Terraform for base infrastructure(including most stateful resources) and CF for stuff like autoscale groups and etc.

Updating GSIs for a DynamoDB table has been difficult for us. We ended up recreating the table in order to change the indexes.