Hacker News Reader

Comment by k__ | Hacker News Reader

How do you deploy NixOS to your servers?

I started a project with nixos-anywhere, deploy-rs, and compose2nix. However, I struggle a bit with secret management.

There are plenty of options: nix-sops, or nix-age, or whatever you would like - past the overall idea the implementation details are purely a matter of taste how you fancy things to be. Key idea is to have encrypted secrets in the store, decrypted at runtime using machine-specific credentials (host SSH keys are a typical option to repurpose, or you can set up something else). For local management you also encrypt to the “developer” keys (under the hood data is symmetrically encrypted with a random key and that key is encrypted to every key you have - machines and humans).

Alternatively, you can set up a secrets service (like a password manager/vault) and source from that. Difference is where the secrets live (encrypted store or networked service, with all the consequences of every approach), commonality is that they’re fetched at runtime, before your programs start.

I’m currently using deploy-rs, but if I’d redo my stuff (and the only reason I don’t is that I’m pretty much overwhelmed by life) I’d probably go with plain vanilla nixos-rebuild --target-host and skip any additional layers (that introduce extra complexity and fragility).

Just a crude and somewhat haphazard summary but hope it helps.

I have been using nixos-rebuild with target host and it has been totally fine.

The only thing I have not solved is password-protected sudo on the target host. I deploy using a dedicated user, which has passwordless sudo set up to work. Seems like a necessary evil.

> I deploy using a dedicated user, which has passwordless sudo set up to work.

IMO there is no point in doing that over just using root, maybe unless you have multiple administrators and do it for audit purposes.

Anyway, what you can do is have a dedicated deployment key that is only allowed to execute a subset of commands (via the command= option in authorized_keys). I've used it to only allow starting the nixos-upgrade.service (and some other not necessarily required things), which then pulls updates from a predefined location.

I do this to remote deploy and it works fine even from my mac

> nix run nixpkgs#nixos-rebuild -- switch --flake .#my-flake-target --target-host nixos@$192.168.x.x --sudo --ask-sudo-password --no-reexec

I'd recommend sticking with deploy-rs. Saved me quite a few hours through its magic rollback which aborted an upgrade that borked VPN.

I strongly recommend investing in some lights-out management (IPMI, KVM or alike) solution that doesn’t depend on any OS peculiarities.

Configuration switching and rollback mechanisms aren’t exactly reliable with trickier setups, as it doesn’t account for any ephemeral state (like what’s actually in the routing tables), and that stuff cannot be always preemptively declared upfront. I’m afraid that despite a lot of efforts, the only truly reliable method to ensure system will come back is still to deploy-and-reboot.

True, it's not a full replacement for ipmi-ish tech. There are ways the deployed waiter can fail before it times out and triggers the rollback. Deploy often enough and you will hit all wonderful edge cases. I treat it as a first line of defense that saves me time on scooting the chair over to me server cabinet and yanking the cords.

WRT ephemeral state -- NixOS allows to minimize this. Coupled with impermanence, all non-declared and non-externally-retrievable state is wiped away upon reboot. And if it's not declative and not retrievable, I just don't use it. Homelab allows for a lot of choice in that regard

I struggled with remote deployment + secret management, too. Like a lot of folks, my nix-config grew over the years as I added secrets management, user management etc ad hoc.

I recently found clan.nix [1] and am quite pleased. It's kind of a framework for writing nixos configurations with a focus on multiple devices. It bundles secrets management and remote deployment into a convenient CLI.

It has the concept of "services", which are e.g. used for user management and VPNs. Services define roles, which can be assigned to machines, e.g. the wireguard service has a controller and a peer role. That feels like the right abstraction and it was very easy to set up a VPN with zerotier like that, something I struggled doing myself in the past.

It's a rather young project, but I converted my nix-config repo after a short evaluation phase to use clan. It's worth taking a look for sure.

[1]: https://clan.lol/

I would strongly recommend sops-nix[0]. Pair this with ssh-to-age/ssh-to-gpg for the keys for each server. We are using this at $work for multiple servers, one notable advantages is that it works in teams (for multiple people) and git (and also gitops).

For remote installations nixos-anywhere is great. deploy-rs or colemna is fine, nixos-rebuild with `--target-host` is also working well for us however.

[0]: https://github.com/Mic92/sops-nix/issues

I seemingly went what I consider simple, others might consider barebones: I have a Makefile, define all the hosts there, and then use scp to copy over all files into the right hosts, and then run `nixos-rebuild switch` on them once copied over. Some have extra "restart-service-X" as some services have their configuration managed outside NixOS (but still in my source repository) so those as well are scp'd into the right place, then the service restarted after the nixos-rebuild switch. Otherwise it's all .nix files and service-specific configuration files.

nixos-anywhere to create, deploy-rs to deploy. sops-nix for secrets. Works great.

Reply on news.ycombinator.com