I assume it's mostly in the field of ... "you're making a semi-large investment on this enough that you're doing semi-custom kubernetes deployments with custom containerd?"
Or maybe the thought is nix-snapshotter users are running k8s/kubelet with nixos anyway so its not a big deal to swapout/add containerd config?
For other distributions, nix-snapshotter works with official containerd releases so it’s just a matter of toml configuration and a systemd unit for nix-snapshotter.
We run Kubernetes outside of NixOS, but yes the NixOS modules provided by the nix-snapshotter certainly make it simple.
Can one even get away with abusing DaemonSets/hostDir/privileged on hosted clusters to modify their own installations? Or is `nix-snapshotter` just sort of out of the question on those provided solutions?
I’m not sure what you mean by modifying your own installation. Like running k8s on NixOS and then using a nix-snapshotter based DaemonSet to modify k8s on the host? At first glance it seems like vanilla k8s can do this already, nix-snapshotter just makes it more efficient / binary matching.
Some folks used to use DaemonSets + hostDir to do that node configuration instead of SSH. Which is weird, but less weird than "you can't autoscale nodes anymore because you have a manual bootstrap step".
Or am I just absolutely missing something?/