Saying ^ because it sounded like you’re planning to use it in “production” as you say small SaaS, which implies your users rely on it and probably pay for it.
It’s often best to start simple - Load Balancer + web/app servers + Database server and add more complexity /features iteratively as you see a real need for it. You’d be surprised how far you can get with achieving K8S perks without bringing in the whole thing - rolling upgrades etc.
Just my 2 cents.
Some notes:
Using helm and helmfile https://github.com/helmfile/helmfile for deployments. Seems to work pretty nicely and is pretty flexible.
As I’m using a consumer internet provider ingress is done through cloudflare tunnels https://github.com/cloudflare/cloudflare-ingress-controller in order to not have to deal with ip changes and not have to expose ports.
Persistent volumes were my main issue when previously attempting this, and what changed everything for me was longhorn. https://longhorn.io Make sure to backup your volumes.
Really hyped for https://docs.computeblade.com/ xD
I use a mix of terraform, ansible, and kubernetes. I like k3s.
In terms of hardware it’s a mix on on-premise, expensive clouds and cheaper clouds. Provisioning new hardware varies between editing a few lines in a terraform file to talking to a salesperson, waiting days, and physically installing things in a servers room. I much prefer the terraform approach.
Consumer hardware like gaming computers can have a great value and last long enough if you are lucky or good while selecting the components. But I wouldn’t put my SaaS on such hardware. Or at least not the critical workloads.
Overall I wouldn’t recommend a small business to bring everything in house. Engineers at AWS, Microsoft, Google, or Hetzner are kinda better and more experienced than you will ever be to manage computers. But if you find it fun and have enough spare time for that, why not for non critical workloads.
1. A cluster VPC with public and private subnets.
2. A managed control plane on EKS.
3. You can opt between self-managed and managed nodegroups. To be fair, self-managed nodegroups give you more control, but you'll need to be careful about updating them yourself.
4. Use AWS controllers - the AWS Ingress Controller, the EBS CSI Driver - stuff like this will ensure your cluster can provision and manage load balancers, storage and so on.
5. For starters use cluster-autoscaler to dynamically manage compute capacity on your nodegroups. As your platform builds up and scales, you can look at Karpenter as a viable alternative.
[0]: https://vmarchaud.fr/deploying-kubernetes-the-hard-way/
This covers a lot of ground, but I don't know if lessons can be ported due to difference in footprint.