How do you manage your CloudFormation processes?
Some common problems I struggle with in AWS CloudFormation:
- Only 1 set of changes for a stack can be run at once. This means if multiple devs are deploying changes then they have to wait and incorporate previous changes before they can provision their templates.
- Not more than 1 person can be testing on staging at once. If Dev A successfully provisions resources and Dev B didn't pull new changes then Dev B may undo Dev A's work. This commonly happens in staging.
- There's so much iterative work required to test if a template provisions everything successfully. What may work in a sandbox may not work in staging. Testing in staging seems necessary, but leads to the previous problems.
- Some rollbacks around ECS may hang for hours before timing out. Thankfully, this has only happened in staging, but I don't believe there was a kill switch? I think there was a way to stop the update from the AWS console, but granting access to that defeats the purposes of having a CI to manage your provisioning.
Thoughts and suggestions?