I'm a Rails monolith and DB migrations are not well suited in ElasticBeanstalk when running docker containers. There's no way to run a single container that 'completes' and then returns whether it was successful or not.
Currently I have a separate environment that holds a single EC2 (lots of idle time) to run migrations. It fires and the deployment process moves on. I have to hope the migration finishes before the next set of containers gets rotated out, 2-3 minutes.
That's not a good policy to run a prod env by so I'm switching to Fargate where I can run a One Off task and poll for the result.
Other benefits include not paying for that dedicated EC2 idle time. Not worrying about EC2 management. Direct access to parameter store and secrets (EB can't do secrets from Dockerrun.aws.json yet =(
Drawbacks - I lose 'Rolling Update/Deployments based on Health' and HealthD. Another big one, I lose access to the container unless you run sshd itself (don't do it).
I am running both environments side by side to evaluate but realistically the migration requirement is going to push me to a full Fargate.
I could only run the migration step in Fargate while the rest in EB - best of both worlds? But somehow that feels dirty... we'll see what I end up doing.
Github Actions is CI/CD and it can be used to kick off any deploy step and manage it, but it's not one to actually perform the migration. That would be like Github Actions actually running the webserver directly.
In this example Github Actions needs to kick off a `eb deploy`, which it can totally do (I personally use CircleCI) but `eb deploy` (elastic beanstalk in general) is not designed to run a short lived script on a single container AND wait for it to return.
You can run a short lived script no problem, but you have no idea if it was successful or not because once the container becomes 'healthy' the command completes.
What I'm really waiting for is the container.lastStatus:'STOPPED' and exitCode: 0. Can't do that on EB.
1) having our app containers include the DB migration logic 2) on container startup, “check and run migrations” before app startup, 3) the trick: acquire a lock in postgres as part of step 2, so that only one node at a time can run migrations.
Migrations are run inside of a begin/commit, so with the lock we have reliable guarantees that 1) exactly one container at a time can try to run the migration; 2) the migration either completes or fails.
This setup has the benefit too that, if the migration fails for whatever reason, no new app containers will start in production. That is: we can basically trigger a production deploy and if it fails, the deploy halts and the previous app version remains up and serving traffic.
There are better ways of handling this, but for us at our scale, this has been both very simple and very reliable, which makes me happy. :)
Then there's issues with when it goes wrong. Troubleshooting is really hard, and knowing what actually went wrong requires a ton of digging around, and god forbid the environment gets stuck in a warning state. You can only download logs when it's "OK", so you're pretty much SOL if this happens. There's also the issue where you can't be sure what's running anymore. If a rollout didn't finish, what actually got updated? There's just no good way to find out, which is absolutely terrible for a deployment system.
All in all, the drawbacks + lack of options/control make EB useless for a lot of companies. Personally, I find it best to avoid EB and just go with tried and tested methods without it. That doesn't mean rolling out K8S or using fargate either. You can if you want, but there are a whole lot of other options.
Terraform is a nightmare to learn for the first time but once you've wrapped your head around it, it's a thousand times better than EB. If Terraform is too complex for your needs then use CloudFormation directly instead of EB.
* It's more cost effective to place all the services on one cluster. As containers can share the instance resources it's easier to increase resource utilisation (not true w/ Fargate though).
* Support for new features on EB can be quite delayed. Eg. we need support for ALPN policies, it's a recent feature and it's not even in CloudFormation yet. With ECS we just manage the LB directly and we can do everything the API allows us to do.
* More granular control. Eg. during rolling updates we can decide a floor to how many containers are active at a given time (many others things like this).
* Integration w/ SSM and Secrets Manager.
* Better IaaS support (with EB everything happens in configuration files outside of Terraform or Cfn resources).