Flowchart: How should I run containers on AWS?
vladionescu.me
vladionescu.me
Honestly this flowchart is kind of a mess, and I certainly wouldn’t recommend it to anyone.
Running nomad, vault, consul together isn't difficult, and will get you a better understanding how to deploy 12 factor apps with good secret storage and service discovery.
Add in Traefik for routing and you've got an equivalent stack to k8s, but you can actually understand what each piece is doing and scale each component as needed.
If you're going to run this all on AWS you can stick to just EC2 and not have to drown in documentation for each new abstraction AWS launches.
As an added bonus nomad can run far more things than just containers, so you have an on-ramp for your legacy apps.
https://news.ycombinator.com/item?id=29063026
Then reading back their vague statements like "A core system in our infrastructure became overwhelmed, prompted by a subtle bug in our backend service communications" and seeing if that matches. It seems to.
But, yes, I'm guessing.
If you don't have the people to do k8s then stick to Lightsail, don't do containers poorly just because you can.
Half-assing it will just make everyone miserable and end up with a mix-match of "thing that can run on ECS" and "things that can't because they need X" where X is really common/useful things like stateful volumes (EBS).
It doesn't seem any harder than EKS, and it's mostly cheaper.
I also find some comments on this article about vendor lock-in dubious, because in the end it's a bunch of containers created from Dockerfiles, which you can easily reuse elsewhere.
I've been running containers for many years on Fargate now and they literally never fail or give me any problems.
Kubernetes on the other hand, weird issues of complexity all over the place. It can do more than Fargate but do you need it? If not, skip it.
This is more or less the equivalent of mandating that everyone use VMS once UNIX started to commodify, or use Windows for all of your servers once Linux has taken over the market. Using EKS + Fargate instead of ECS + fargate provides no savings over EKS, All it serves to do is to lock you into a single hyper-scaler infrastructure, at the same time as K8s is forcing he cloud vendors into commoditization.
Want to use AWS effectively? Depend on high level services like Glue, Athena, Kinesis Firehose, Sagemaker. Want to piss away any chance to run your business effectively? Leverage ECS.
If you are a one-man shop, and you know ECS or are willing to depend on underbaked solutions because they solve a problem, more power to you. I suspect that you may benefit over your career by investing in more universal skill sets (alternative, you may benefit from hyper-specializing on AWS toolsets as well(.
AWS is good at giving you commodity infrastructure at scale. They are really good at "stupid" services like S3 or EC2 or RDS. The high-level services meanwhile? I worked with quite a few of them and they're mostly shit.
Athena is has account-wide limit of 5 concurrent queries which cannot be increased. Even one larger dashboard will overload it.
Redshift have the same limit of 30 concurrent queries. That's good enough for casual use but not suitable for larger company.
Glue Catalog does not scale at all, and having more than few objects will break it, and you will very soon end up begging for more API limit due to throttling exceptions.
Kinesis has very strange limits (in messages per second) that make it really expensive for use cases where traffic is peaky - which is quite a few streaming use cases.
Just like you wouldn't use Amazon.com to buy high-quality, important goods (because you're pretty likely to get something broken or fake), don't use AWS "high level" services. Amazon is company focused on scaling commodity use cases, not at engineering excellency.
We asked for that as well, and at first the support said they did that, but turns out they increased the Athena queue length instead of concurrency - so the queries wouldn't fail (until the queue is full, anyway) but the throughput was still the same.
Then the team of AWS account managers and solutions architects (who assured us many times that the concurrency can be increased) get into a email discussion with the support and Athena team, before coming back and telling us it cannot be increased. No apology or anything. Recommended us to run Presto on EMR instead.
It was a sobering experience in how AWS operates.
I think of it slightly differently. S3 is clearly the product of engineering excellency (as is DynamoDb).
But I think there are A teams and B teams at AWS, as you would expect from any company so large. And I get the sense that at least some of the newer, less proven solutions were written by the B teams. This likely includes some k8s-market-share-grabby offerings.
But that would not be software they build "for someone else", like Redshift and Glue and Sagemaker and whole lot of other stuff. There's a whole lot of half-baked services in the ~200 they have.
I've used Fargate as I thought it was the easiest/cheapest route, but according to the notes on chart:
"From 2020-2021 the best option was/is ECS and Fargate".
"Maybe from 2023-ish AppRunner will become the best option. It's a preview service right now but on the path to be awesome!"
"It may not totally take over ECS on Fargate, but it will take over most common usecases."
Agreed with the other commenters that this flowchart is not to be recommended, for this and other reasons.
(Look at where the top of the "sustainable zone" is...5%? Like 1/20th of a vCPU?)
I still need to explore how AppRunner compares to CodeBuild for this purpose.
For use-cases like distributed job scheduling, an EventBridge event triggering either an AppRunner request or a CodeBuild execution or an ECS task could work. It's still a little tricky to figure out what the best choice is.
It comes to the same conclusions that I intuitively had as well:
- EKS if you really have to use k8s - ECS if you have a modicum of complexity but still want to keep it simple - AppRunner if you just want to run a container
For those not in the know, ASG based Capacity Providers are hard to work with because they are very immutable so you end up having to create-then-delete any changes that touch the capacity provider. A capacity provider cannot be deleted if the ASG has any instances. Many tools like terraform’s AWS provider will refuse to delete the ASG till the capacity provider is deleted. The terraform provider just cannot properly reason about the process of discovering and scaling in the ECS tasks on the provider, scaling in the ASG, waiting for 0 instances, and then deleting the Capacity Provider. It’s honestly beyond how providers are supposed to work.
TL;DR: The flow chart is somewhat correct: Do everything in your power to run on ECS Fargate. It’s mature enough and has excellent VPC and IAM support these days. Stay as far away from ECS on EC2 as you can.
As for EKS, I like it but this company runs on the whole “everyone can do whatever” so each team would have to run its own EKS cluster. If we had a centralized team providing a base k8s cluster with monitoring and what not built in for us to deploy on, I’d be more amenable to it. As it stands, I would have to learn both the development and ops AND security sides of running EKS for a handful of apps. ECS while seeming similar on the surface is much simpler and externalizes concepts like ingresses and load balancing and persistence into the AWS concepts and tooling (CDK, CloudFormation, Terraform) you already know (one hopes).
Importantly both of which are almost impossible for "normal" developers to use with any level of competency. This leads to 2 inevitable outcomes, a) snowflakes and all the special per-app ops work that entails and b) app teams pushing everything back to infrastructure teams because they either don't want to work with TF or can't be granted sufficient permissions to use it effectively.
k8s solves these challenges much more effectively assuming your ops team is capable of setting it up, managing the cluster(s), namespaces, RBAC, etc and any base-level services like external-dns, some ingress provider, cert-manager, etc.
Once you do this then app teams are able to deploy directly, they can use helm (eww but it works) to spin up whatever off-the-shelf software they want and are able to easily write manifests in a way that they can't fuck up horribly as easily.
Best for both teams, ops and devs. Downside? Requires competent ops team (hard to find) and also some amount of taste in tooling (use things like tanka to make manifests less shit), not to mention the time to actually spin all this up in peace without being pushed to do tons of ad-hoc garbage continually (i.e competent org).
So in summary, k8s is generally the right solution for larger orgs because it enforces better split of responsibilities and establishes a powerful (relatively) easy to use API that can support practically everything.
Also in the future there is things like ACK (https://aws.amazon.com/blogs/containers/aws-controllers-for-...) coming which will further reduce the need for app teams to interact with TF or Cloudformation.
And therein lies the problem, my employer won't consolidate the enterprise like that. They're still throwing 100 or so teams around and telling them BIRI (build it, run it) and making everyone write their own infra and application code because "DEVOPS".
Makes me think there is probably a market for a k8s w/batteries distribution + ops as a service available via AWS VPC peering/transit GW if such a thing doesn't already exist. i.e pre-setup with good patterns, pluggable/easy OIDC auth, cert-manager and friends.
Essentially to cater to the "I need k8s but I don't have the people for it and this makes me sad" crowd.
Even when running non-reserved EC2 instances to make up our ECS cluster, it is cheaper than using Fargate.
That said, moving containers to another container orchestrator isn’t terribly difficult, so I don’t personally worry about vendor lock-in for containerized workloads. If your workloads have dependencies on other vendor-specific services, that’s a different story, but basically a container is easy to move elsewhere.
We also chose our current cloud providers because the alternatives were worse.
The cost to move your operations expertise to another platform and learn all of its new quirks might be significant though.
It’s that simple. If you need extensions into the AWS infrastructure, check out their CRD extensions that allow you to provision all of the infrastructure using K8s.
Edit: Turns out I read it wrong, apologies. This guide is for companies with less than 250 engineers.
I'd guess that if your org is big enough to have 250 engineers in its payroll, AWS services are a waste of cash given you can deploy better and cheaper. For example, Hetzner has less than 200 employees, and it's a global cloud provider.
The bill is not only astronomically high but it's also unpredictable and uncontrollable. Which added valued do you believe justifies this?
Meanwhile, keep in mind that the likes of Dropbox gave AWS a try but in the end learned from their experience and opted to migrate out after only 5 years.
> There’s a restrictive factor when hiring if you move away from Google Cloud,Azure, and AWS.
I disagree, considering that in AWS you either run vanilla VMs from EC2, or containerized solutions in ECS/EKS. Either way you are already better off using either your local VMs or your company's kubernetes deployment.
If you've got 250 engineers then the AWS is going to be fairly predictable and well monitored.
>Meanwhile, keep in mind that the likes of Dropbox gave AWS a try but in the end learned from their experience and opted to migrate out after only 5 years.
Dropbox is memorable because they migrated off which says what many other companies don't do.
As you can tell by the massive growth of Azure many large enterprise companies definitely see the value despite already have their own datacenters.
This is the very first time I hear anyone describe AWS services as "fairly predictable and well monitored", because there is simply no such thing. There are professional services being sold at a premium whose value proposition is to make sense of AWS billing and reign it in. With AWS, your company simply sets a budget with a significant slack and hope for the best.
> Dropbox is memorable because they migrated off which says what many other companies don't do.
Not true. Dropbox is memorable because it was depicted as a poster child of AWS, which ultimately found out it was not worth it at all. And Dropbox is not alone.
https://www.datacenterknowledge.com/archives/2016/03/16/movi...
https://www.uscloud.com/blog/3-reasons-33-companies-ditching...
Enterprises provided many of the things inside AWS but much, much worse.
I actually think the political revolution it (cloud) triggered was a bigger thing than the technical and engineering changes.
AWS is expensive compared to a machine under your desk but not so much compared to the fully loaded cost of (say) a VMware VM at a bank DC, which includes multi-year enterprise hardware & software commitments to make it go plus staff and admin costs.
The DC model choked off innovation that was not approved "from the top" and severely damaged agility.
All major cloud providers have similar computation pricing. As much as I love Hetzner, they are not a major cloud provider.