> Could you elaborate a little on this part? Our team is looking at ECS/Fargate as a possible container solution and I am curious what you felt was missing from it.
Some typical examples:
- Kubernetes allows you to run a monitoring container on every single node using something called a "DaemonSet". On ECS, you'll have to build all your monitoring tools into your base image, or use cloud-init to spawn an ECS task on each machine.
- You're probably going to end up writing a bunch of scripts to generate ECS task definition JSON and to update running services, and you'll need to integrate this into your CI system somehow. With Kubernetes, you can get away with "kubectl apply -f" for a fairly long time.
- Kubernetes makes it relatively easy to allocate and manage persistent disk volumes. I wouldn't necessarily use them for a production database, but they're great for smaller things.
- Kubernetes has autoscaling support, plus the ability to control which containers run on which types of servers.
- Kubernetes has basic secret management built-in. It's nothing as nice as Vault, but it's good enough to get started.
- Kubernetes has support for a whole bunch of useful minor things that would typically wind up as Terraform scripts on AWS.
And so on. None of these is very hard individually, but there's a ton of things like this. So we're slowly migrating pieces of ECS infrastructure over to Kubernetes so that we can stop reinventing so many wheels.
Again, for those things which can run on Heroku, either choice is overkill. And I have to admit that ECS is very reliable at the things it does. If you do decide to look at Kubernetes, I highly recommend skimming the O'Reilly books, which provide a solid overview of how it all fits together.