Show HN: kproximate – A Kubernetes node autoscaler for Proxmox | Hacker News Reader

Show HN: kproximate – A Kubernetes node autoscaler for Proxmox

github.com

60 pointssomeofmyparts27 comments

I wonder how hard it would be to port this over to a Karpenter variant. I suppose the Proxmox API won't do the same things as an AWS EC2 API so an operator-in-the-middle would still be required.

Won't this just result in over-committing the physical nodes? Or am I missing something?

Host resources might and should be overprovisioned, this ensures the k8s clusters you run on proxmox aren't overprovisioned.

If your k8s nodes are the same height and width as your Proxmox servers, then yes. In my case, I have clusters of ~7 nodes, 100 cores, 1.75TB RAM, but my VMs tend to be much, much smaller. My web workers are each a handful of cores, I could double them a couple times before saturating the physical nodes core count.

Wow that is some serious horsepower.

I guess I just don't get why you'd have such small k8s nodes in a cluster, I have a cluster about half that size (in node and core count) but just run k8s directly on the metal. It means that things are quite elastic with an HPA + VPA on things that need it. I feel like we get better overall utilization of the hardware, but I'd be interested in hearing about some of the reasons for your approach.

Off the top of my head, there are probably a couple of 'pros' to doing it your way; one: experimentation is probably easier. Two, I'm willing to bet networking isn't anything too fancy (getting k8s to play nice on bare metal networks is a shitty thing, especially once multiple NICs are involved -- a LOT of networking stacks, ok, all of them, just 'assume' that the first network card it sees is THE network card it should use... and don't even bother to 'sort' them so its the same order every reboot).

selfish question but I'm about to give talos/cilium a go, have you evaluated that? my smaller testing was encouraging

Yep. Running Cilium in raw L2 mode (no encapsulation) on the lan, and even using external IP's in L2 mode to provide load balancing (currently a beta feature, and switched from MetalLb). It has worked beautifully with very few issues.

If you're running k8s on Proxmox, it's probably because you aren't 100% k8s yet, or are a homelab and experimenting with many things.

What is the benefit to running multiple Kubernetes nodes on one physical Proxmox machine?

Thinly provisioned compute in the presence of other VMs is one reason, but I think this is aimed at people with multiple proxmox nodes in a cluster.

> Thinly provisioned compute in the presence of other VMs is one reason

That's the only reason I can think of.

> I think this is aimed at people with multiple proxmox nodes in a cluster

I think so too, but then we're back to the original question again (i.e. why not just have kubernetes nodes instead of having proxmox nodes with kubernetes VMs on them).

If you host multiple proxmox nodes, you can host other VMs on each bare metal node in addition to the kid cluster, right?

Yes. Seems reasonable if you have much more compute than you can saturate with just things in your kubernetes cluster. Otherwise just make those hosts kubernetes nodes instead and call it a day.

Why make pets when you are already set up for cattle

A kubernetes worker node (on bare metal) is not more of a pet than a proxmox node IMO. It's just an unnecessary layer if you're gonna fill it with kubernetes pods anyway. Again, only if you are not planning to run non-kubernetes VMs. If you are then Proxmox with kubernetes VMs seems good.

You can also do things like live VM migration, where the K8s nodes get transferred to a different Proxmox node without interruption.

Arguably niche, but if you have a small cluster (I have 3 physical nodes in my Proxmox cluster at home), it's sometimes nice to be able to just shift everything to do maintenance without having to worry about losing quorum or disrupting workloads.

True! Didn't consider that.

Something a lot of other comments seem to be skipping out on is storage, proxmox provides an easy hyperconverged option with its support for CEPH, which provides a much more battle tested option when compared to k8s solutions like longhorn.

Good point

Failover, green-blue upgrades, experimentation are 3 things that come to mind.

Proxmox can also automatically provision zfs underneath, automatically set up ceph across nodes, has a nice ui for the firewall and more. Maybe the benefit is convenience.

Infrastructure and devops becomes more abstracted with every new word salad.

I am a little tired of commenters lazily flinging this every time something in the infrastructure world comes up that they don't fully understand. Something doesn't become word salad just because you don't know what it means.

I might have problems understanding something -- say, "Comparison of implementations of in-order traversal of an AVL tree in Scala v/s Clojure"; but that doesn't mean it doesn't have a clear and intelligible meaning to someone who gets it.

In this case, I understand the idea here somewhat even without looking at the project. Kubernetes has very flexible auto-scaling capabilities. This seems to be be somehow allowing it to scale across a bunch of VMs hosted using Proxmox, which is an open-source VM hosting platform. There is no word salad about it.

Agreed, the title was pretty self explanatory.

I have heard of Proxmox but didn't really know what it did, so I went to Wikipedia, which tells me: "Proxmox Virtual Environment (Proxmox VE or PVE) is a hyper-converged infrastructure open-source software."

This is a little buzzwordy. I am also not sure that something can be "a software".

English is not my first language but i think the "a" is for the infrastructure. It's software for a hyper-converged infrastructure.

If you remove the "a", to me, it sounds like the software is hyper-converged instead. Which seems less correct.

Anyway I agree. The very next couple of sentences describe it better in my opinion. Maybe they should put those two first:

"It is a hosted hypervisor that can run operating systems including Linux and Windows on x64 hardware. It is a Debian-based Linux distribution with a modified Ubuntu LTS kernel and allows deployment and management of virtual machines and containers"

Reply on news.ycombinator.com