Hacker News Reader

Comment by jrochkind1 | Hacker News Reader

Thank you for this!

What in your opinion is the optimal way to run "a couple dozen ec2 instances"?

I'm kind of in that zone.

I don't think "totally manually" works out very well once you are in even "one dozen EC2 instances", it's too easy to make mistakes, you spend too much time (with downtime) figuring out what mistake you made and what it was supposed to be instead. Generally too hard to orchestrate changes without downtime.

But is there anything in between totally manual and kubernetes?

I think that's one reason people are reaching for kubernetes, they actually are having challenges that need a solution, and can't figure out what the solution is other than kubernetes.

View on HN

It’s unpopular, but I embrace (AWS) lock-in for small and mid-sized infrastructure. AWS’ various services solve 99.9% of any infrastructure needs. High-bandwidh streaming and heavy gpu processing is one of the few places my customers find to be cost prohibitive.

For compute I lean towards an immutable infrastructure. I was a Puppet / Chef engineer for 8 years, never again. It’s cleaner to configure apps via some mixture of pre-baked amis, docker files and some sort of parameter store. (Usually SSM or Vault).

Fargate is a cleaner path but so much more expensive.

Thanks for response! Sure, I'm down with AWS lock-in, for the sake of argument.

But how do you manage this EC2 fleet and collection of other AWS services? Changes happen -- new version of your local software, change your mind about some configuration, new version of a database, whatever.

OK, it's immutable. But I still need to launch a new EC2, and get everyone else that's supposed to know about it to know about it and start talking to it instead of the old one (or get public traffic to go to it or whatever), and then take down the old one once it's safe to do so. (I also need to figure out how to construct it as immutable).

Purely manually? Just manual operations in the AWS console?

There may be an obvious answer here I am missing not being familiar with the tools. What can be overwhelming here is figuring out the "right" way to orchestrate this stuff, there are so many options (or sometimes you can't find any options but suspect you are not looking in the right place) and so many partisans. One reason for kubernetes popularity is the succesful awareness marketting, I've heard of it and know it is supposed to solve these challenges I'm having and don't really understand what the options are and am overwhelmed trying to figure it out... but k8 seems popular, it must be good?

Your CI pipeline can build an AMI image with integrated Docker image, and a simple systemd script to start it on boot.

Deploy step would just be to upgrade AutoScalingGroup with the new AMI, in a rolling fashion with CloudFormation.

---

However, in most cases I'd recommend going with ECS + Fargate, where you can just deploy your Docker image, and don't have to care about the underlying virtual machines.

---

If you want to go K8s route, EKS + Fargate, (or if you want to go cheaper, EKS + EC2 Node Groups), are the best way in most cases, but you need to be aware that you will have to keep an eye on upgrading EKS versions, Helm charts, and other resources you add to the cluster.

I'd say Kubernetes is a good tool if you need it, but otherwise I'd start with something that doesn't require any maintenance on the ops side.

This. I'm not some Unix sysadmin type who wants to do everything manually. But the AWS features work out of the box for basic deployment, scaling, etc. and Kubernetes seems to just add unnecessary operational complexity without any value if you don't need any other features and are okay with AWS lock-in.

The canonical AWS answer here is: Elastic Beanstalk.

If you use a CI tool, it probably has a CD counterpart which can pipe a passing build directly to EB.

Or you can deploy manually from your repo with "eb deploy".

In either case, AWS will automatically bring the app up on new instances, and test them. If everything looks good, the load balancer will direct live traffic to the new instances, and then spin down the old ones.

Config changes use a similar process.

Why not use AWS's managed k8s and eliminate needs for your job? I can't actually think what k8s consultant would do with managed k8s.

Obviously a personal opinion, but Ansible solves this problem really well. It does get sluggish once we're talking about more than 50 machines, though.

To reliably end up with a working cluster:

- spin up plain operating system VMs

- spin up one more VM for Ansible instigator

- collect the IPs, write them down to Ansible inventory

- pull/upload your custom binaries

- run Ansible playbooks to set the VMs up

Sometimes it's beneficial to have a two-step process. One playbook to install rarely changing components. Then make a VM snapshot. The second playbook installs your custom binaries. Future installations can then start from the snapshot.

There are lots of ways to skin the cat.

Back in the day I worked on code to create custom Linux ISOs with network provisioning. Always wondered why more people didn't do this. You just need a kickstart script. With a simple provision script you can have a fully automated install. No messing about copying ssh keys or setting IP addresses, etc.

Reply on news.ycombinator.com