I've used Ceph together with Proxmox VE excessively. No problems whatsoever.

And in related news, Proxmox VE is often probably a more sensible thing to use for a private cloud environment, because it is far more flexible and easier to use than Kubernetes.

For small setups it’s honestly fine with rook. For large ones yeah better dust off your ceph phd

as much as i'm glazing k8s in this thread I haven't managed to get ceph working. I wish it too since I dont want to use minio anymore.

Longhorn just kinda worked out of the box though with a couple kernel/system settings. No s3 api though.

But this isn't k8s fault out all.

The root cause here is just that managing any kind of storage service is instantly painful. The property of "not losing data" means that you are sort of required to always be doing something in order to keep it healthy.