Hacker News Reader

Comment by Asdrubalini | Hacker News Reader

My router (a VM with passed through network card) is currently running on NixOS, I’m really happy so far, it makes kernel patching (which in my case is required for the nic to work properly at 2500BASE-T) much easier than manually patching and building the kernel every time. It also gives me the guarantee that I will be able to rebuild the exact same host almost bit by bit in the future, in case of necessity, which is important for something like a router (I also take vm snapshots but upgrades are crucial for something potentially facing the internet and handling firewall). Looking forward to move away from proxmox to a fully NixOS virtualisation host too.

View on HN

Yeah I used to use pfSense but hated having a load of config and firewall rules created manually, it all felt quite brittle.

Moved to NixOS almost 5 years ago now and haven’t looked back, my router config is in Git and I’m able to make changes with confidence.

Best part is making a change/installing an update and being able to rollback if I mess something up. I also use Nixus to push new configs which has a nifty feature where it will autorollback if you e.g make a config change that locks you out of SSH.

I also moved from ESXI to NixOS for a virtualisation host more recently and that has been great as these machines are inherently long lived so you can’t apply the “cattle not pets” mindset here, but I feel like I can make changes and install updates with confidence. It also helped when I migrated to a Threadripper build where there were some strange quirks which needed Linux kernel patches, it’s nice being able to do this declaratively and to store everything in a versioned Git repo.

Wow, didn’t know about Nixus, thanks!

Can you share your router configs? I'd love to have a NixOS based router.

https://skogsbrus.xyz/blog/2022/06/12/router

One drawback I realized about NixOS as a router for home use is that if you need to share administration of the network, then it has to be easy for non-experts. In that case something like OpenWRT might be a better choice.

Well, at least the config file options are relatively straightforward, but you're right in that a GUI makes more sense for genpop

https://gitea.c3d2.de/zentralwerk/network

How are you managing the VM’s on NixOS?

A couple years ago I was searching for a good way to declaratively manage them, and the best I came up with was libvirt + terraform + nixos-generators for images. It feels like you should just be able to set them up as systemd units, but I couldn’t figure it out.

microvm has caught my eye recently but I haven’t played with it yet: https://github.com/astro/microvm.nix

Not OP (and I suspect they don’t have what you’re looking for since they have stated they treat the VMs as pets, not cattle). I currently use libvirt, but I eschew persistent VMs and instead build the libvirt domain XML with Nix, and then start VMs with `virsh create /path/to/myvm.xml`. Using an absolute path means that libvirt will not persist the domain, and makes it easier to declaratively manage the config with Nix.

However, none of my VMs are always-on, so I don’t have to deal with restarts on config change. That said, when libvirt isn’t managing the lifecycle of the VMs as much, writing a systemd unit should be quite a bit easier.

The VMs themselves are a mix of some manually created ones (the VM configs are still NixOS though), and some IaC VMs which I just use Terraform and the libvirt provider. It's a bit clunky but I wouldn't want to embed the VM config in the hypervisor configuration as the VMs can be live migrated to other hypervisors.

MicroVM looks awesome though, thanks for the pointer!

Did you manage to get live migration working with the terraform provider? As of a while ago it wasn’t implemented but idk if that’s changed: https://github.com/dmacvicar/terraform-provider-libvirt/issu...

I appreciate your comment here, and the parent comment.

I've been using OPNSense[0] as my router for the past several months. So far, I am quite happy with it, but I've also thought that NixOS would be the next step.

My OPNSense router has 30+ VLANs and many layered firewall rules (my take on "zEr0 tRuSt") and so the task of converting it all to NixOS seems a little daunting.

I appreciate the utility of OPNSense's web GUI when configuring and troubleshooting my router config. It would be awesome if something like that could be integrated with NixOS. Additionally, something like nsh[1] to provide a traditional router/switch style CLI would be a dream come true.

[0] https://opnsense.org/

[1] https://www.nmedia.net/nsh/

NixOS is great for a basic home router (I use it for my home router) but it’s networking config is still pretty rudimentary, and some things I would expect to work just don’t - i.e. port forwarding only works from outside your network, not inside.

I haven’t done much with vlans yet so I can’t comment on that.

I would highly recommend to use systemd-networkd based networking which should be able to do almost everything if configured correct.

Is there a nix config wrapper for systemd-networkd?

NixOS still does everything I need it too, but the parent comment sounded like they had a bit more complex of a setup.

Not sure what you mean by config wrapper, but yes, systems-networkd is quite nice. Here's my home router setup with vlan1 for uplink to ISP, a bunch of other vlans for internal networks, and IPv6 prefix delegation to internal networks, and of course fireguard. All configured using systemd-networkd:

{ pkgs, lib, ... }:{

  networking = {
    useNetworkd = true;
    useDHCP = false;
    enableIPv6 = true;
  };

   networking.wireguard.interfaces = {
     wireguard = {
       ips = [ "172.20.60.1/24" ];
       listenPort = 61891;
       privateKeyFile = "/etc/nixos/secrets/wireguard-privateKey";
       peers = [
         {
           publicKey = "897mRPejuv9yVnmTvcUL7ckQkIiM0wnSgHmgR15Evyw=";
           allowedIPs = [ "172.20.60.10/32" ];
           presharedKeyFile = "/etc/nixos/secrets/wireguard-presharedkey";
         }

...

  systemd.network.networks = {
    "10-eno1" = {
      matchConfig.Name = "eno1";
      networkConfig.LinkLocalAddressing = "no";
      networkConfig.DHCP = "no";
      extraConfig = ''
        VLAN=wan
        VLAN=vlan99
        VLAN=vlan30
        VLAN=vlan20
        VLAN=vlan1
        VLAN=podnet
        LLDP=no
        EmitLLDP=no
        IPv6AcceptRA=no
        IPv6SendRA=no
      '';
    };
    "11-vlan1" = {
      matchConfig.Name = "vlan1";
      linkConfig.RequiredForOnline = false;
      networkConfig.DHCP = "no";
      networkConfig.Address = "192.168.1.1/24";
      networkConfig.Domains = "tanso.net";
      networkConfig.ConfigureWithoutCarrier = "yes";
    };
    "11-podnet" = {
      matchConfig.Name = "podnet";
      linkConfig.RequiredForOnline = false;
      networkConfig.DHCP = "no";
      networkConfig.Address = "172.20.2.1/24";
      networkConfig.Domains = "tanso.net";
      networkConfig.ConfigureWithoutCarrier = "yes";
    };
    "11-vlan20" = {
      matchConfig.Name = "vlan20";
      networkConfig.DHCP = "no";
      networkConfig.Address = "172.20.20.1/24";
      networkConfig.Domains = "tanso.net";
      networkConfig.ConfigureWithoutCarrier = "yes";
      extraConfig = ''
      IPv6SendRA=yes
      DHCPv6PrefixDelegation=yes
      '';
    };

....

  systemd.network.netdevs = {
    "11-vlan1" = {
      netdevConfig = { Name = "vlan1"; Kind = "vlan"; };
      vlanConfig.Id = 1;
    };
    "11-podnet" = {
      netdevConfig = { Name = "podnet"; Kind = "vlan"; };
      vlanConfig.Id = 2;
    };
    "11-vlan20" = {
      netdevConfig = { Name = "vlan20"; Kind = "vlan"; };
      vlanConfig.Id = 20;
    };
    "11-vlan30" = {
      netdevConfig = { Name = "vlan30"; Kind = "vlan"; };
      vlanConfig.Id = 30;
    };

> Is there a nix config wrapper for systemd-networkd?

The options under systemd.network almost map 1:1 to systemd-networkd ones.

This has more to do with iptables/nftables I think. I solved this by adding my port forwarding rules to the prerouting chain and the output chain. You can use a jump to consolidate the rules, like so - https://gist.github.com/madjam002/d30f6000adf0761e92623f7de2...

As other commentators have said, switching to systemd networkd has allowed for more advanced network configurations than I ever managed with pfSense. IPv6 works pretty much out of the box with my ISP which uses DHCP prefix delegation, I can assign /64's to different VLAN networks, and more recently I set up network prefix translation (I think that's what it's called) for my lab kubernetes cluster so each pod gets a unique IPv6 address in the RFC 4193 range which maps to my actual ISP provided IPv6 prefix, so if I change ISP the IPs in the cluster would remain the same. Being able to expose Kubernetes services directly to the internet with unique IPv6 addresses is pretty nice :)

Port forwarding not working from inside the gateway sounds like it’s not doing hairpin NAT by default. I’m running a NixOS home router pair, not currently doing any port forwarding but now I’ll have to check that tonight, if nothing else people might appreciate an option for it.

Did you base your router config on some existing template or did you just built it up from scratch?

I based much of the configuration on [1] but I took the nftables rules from the official nftables wiki.

[1] https://francis.begyn.be/blog/nixos-home-router

Reply on news.ycombinator.com