Hacker News Reader

Comment by pmig | Hacker News Reader

I agree that ECS works great for stateless containerized workloads. But you will need other AWS-managed services for state (RDS), caching (ElastiCache), and queueing (SQS).

So your application is now suddenly spread across multiple services, and you'll need an IaC tool like Terraform, etc.

The beauty (and the main reason we use K8s) is that everything is inside our cluster. We use cloudnative-pg, Redis pods, and RabbitMQ if needed, so everything is maintained in a GitOps project, and we have no IaC management overhead.

(We do manually provision S3 buckets for backups and object storage, though.)

View on HN

Mentioning “no IaC management overhead” is weird. If you’re not using IaC, you’re doing it wrong.

However, GitOps is IaC, just by another name, so you actually do have IaC “overhead”.

Exactly, not only because Flux/ArgoCD are inherently some sort of IaC themselves, but also because on top of those tools you’ll need to have Terraform to manage the K8s cluster as well as a good practice.

And most GitOps workflows have considerably more complexity and overhead than your typical Terraform "IaC" workflow.

Many companies run k8s for compute and use rds/sqs/redis outside of it. For example RDS is not just hosted PG, it has a whole bunch of features that don’t come out of the box (you do pay for it, I’m not giving an opinion as to whether it’s worth the price)

Yea RDS makes your life easy, notifications and easy application of security patches both OS and DB level (minor version upgrades). Easy upgrade of major versions, easy upgrade of storage, RAM and compute (but not so easy to downgrade), easy options for replication, Blue/Green deployments etc to name a few.

Yup. We do that.

Anything stateful is not allowed inside the cluster. PVs are annoying enough without having to manage a DB bolted onto what was originally designed for stateless web services.

My db is my cluster. It's been stable for years but I'm afraid to touch it. There's a long outstanding issue in k8s that makes PVs harder to resize than it should be. And they're just more complicated. Trying to move to managed MySQL now. It'll cost me a bunch more but at least I get a fail over node which I don't know how to set up myself... Still no master-master though, apparently that's not an option.

Actually resizing PVC depends on the CSI drive. Some support easy resizing, some require the volume to be detached. You can double check your CSI driver and might just need to patch your storage class.

Agreed running databases without operators that can handle replication, master promotion backups and PIT restore is super scary. Most of the modern operators support all of these operations.

In everything you've listed, my conclusion is the opposite. The spread across multiple managed services is not a bad thing, that's actually better considering that using them reduces operational overhead. That is, the spread is irrelevant if the services are managed.

The ugliness of k8s is that you're bringing your points of failure together into one, mega point of failure and complexity.

Final aside - you absolutely should be using IaC for any serious deployments. If you're using clickops or CLI then the context of the discussion is different and the same critera do not apply.

Yes, but our GitOps repository heavily utilizes Kustomize and Flux, allowing us to reuse a significant amount of code across multiple deployment stages and clusters, which has proven to be very effective.

We have worked with Terraform modules before, but they quickly became difficult to manage.

Additionally, deployments to ECS are typically handled by invoking the AWS API within a GitHub Action, without continuous reconciliation or drift detection.

> Additionally, deployments to ECS are typically handled by invoking the AWS API within a GitHub Action, without continuous reconciliation or drift detection.

No they aren’t. All of the major IaC solutions (TF, CDK, etc) do ECS deployments directly through their own API, including with drift detection and updates.

Good for you for finding something that works, but it sounds like your advice related to IaC solutions is based on a misunderstanding of the benefits of IaC and the tools available.

You make a great point that when everything is on kube it’s easier to manage.

But… if you are maintaining storage buckets and stuff elsewhere (to avoid accidental deletion etc, a worthy cause) then you are using terraform regardless. So adding RDS etc to the mix is not as tough as you make it sound.

I see both sides of the fence and both have their pros and cons.

If you have great operational experience with kube though I’d go all in on that. AWS bends you over with management fees… it’s far more affordable to run a DB, RMQ, etc on your own versus RDS, AMQ

AWS Controller for Kubernetes (ACK)[1] provides resources for creating S3 buckets as CR. Also in combination with Pod Identities there is no need for tf.

[1]https://aws-controllers-k8s.github.io/community/docs/user-do...

You’ve replaced IaC overhead with k8s overhead

How do you run all this on developer's machine?

Reply on news.ycombinator.com