Understanding modern Linux routing (and wg-quick) | Hacker News Reader

Understanding modern Linux routing (and wg-quick)

ro-che.info

179 pointsEntICOnc28 comments

> Older VPN scripts used to override your default route in the main table when connecting to the VPN and restore it when disconnecting. Sometimes this wouldn’t work, and after disconnecting from the VPN you would be left without any default route at all. wg-quick doesn’t have this problem, as it never messes with your main routing table. All it has to do when disconnecting is delete its two rules, and your default route is active again, Or you could even do that yourself with ip rule del.

Isn't this the same problem the old VPN problems had though? If it fails to delete its two rules then your routing is still boned.

For comparison OpenVPN doesn't use the ip rule infrastructure, instead it installs two rules directly on your routing table, but not for 0.0.0.0/0. Instead it installs rules for 0.0.0.0/1 and 128.0.0.0/1, creating a rule that has slightly higher priority than the default route so it can leave the default in place and not get burned with NetworkManager decides to delete and re-add the default route for whatever reason.

Not having the routes cleaned up after the VPN exits is usually not the problem, as they are tied to the TUN interface and get cleaned up by the kernel when that interface is destroyed anyway. The bigger issue is clients that forget to install the host route to the VPN endpoint. I see that error a lot and it always drives people crazy because their VPN client negotiates and connects properly, and then everything breaks for no obvious reason.

Yeah, other VPNs do the same thing as wg-quick. Although they also may make a ton of more specific routes which take precedence over ones from wg-quick. It's fairly rare that routes don't get cleaned up, but a reconnect/disconnect or a reboot fixes it.

The host route to the VPN endpoint is sometimes there, but sometimes it's to a p-2-p interface, sometimes just the local device IP. Depends on the OS, tunnel interface, routes, etc.

>"For comparison OpenVPN doesn't use the ip rule infrastructure, instead it installs two rules directly on your routing table, but not for 0.0.0.0/0. Instead it installs rules for 0.0.0.0/1 and 128.0.0.0/1, creating a rule that has slightly higher priority than the default route ...."

I'm curious is the priority you referring to just longest prefix match by having two /1s, or are you referring to something else like setting a metric on the interface?

Yes, they are more specific routes than 0/0. In Linux can also have a metric, so if two routes are equal one can be preferred over the other. On networking devices you usually have administrative distance (different routing protocols) and metric (within one routing protocol).

Yes I understand how routing works and I referenced both of these things in my question. I was asking specifically which one the OP was referring to.

Wasn't it obvious what OP meant then?

> Instead it installs rules for 0.0.0.0/1 and 128.0.0.0/1, creating a rule that has slightly higher priority

More specific prefix isn’t slightly better, prefix length is one of the most significant rules of routing. This trick is quite common in networking - you point more specific routes to the main link and less specific to a backup. It the main link go down the specifics are getting withdrawn (some mechanism like dynamic routing should be in place for that) and the less specific route takes place, redirecting the traffic over the backup link.

I’ve done this myself many times (network engineer).

But tbh I see it as something of a “hack”. My gut tells me it’s not the right way for something like OpenVPN to do by default.

A metric would be better on 0/0, but I can see how the two /1 routes is gonna fix things in a whole bunch of edge cases.

>> Older VPN scripts used to override your default route in the main table when connecting to the VPN and restore it when disconnecting. Sometimes this wouldn’t work, and after disconnecting from the VPN you would be left without any default route at all.

> Isn't this the same problem the old VPN problems had though? If it fails to delete its two rules then your routing is still boned.

You're right, that's not the reason.

The real reason WireGuard doesn't use the traditional technique of replacing the default route is, straight from the horse's mouth https://www.wireguard.com/netns/:

  The most straightforward technique is to just replace the default route, but add an explicit rule for the WireGuard endpoint:

  # ip route del default
  # ip route add default dev wg0
  # ip route add 163.172.161.0/32 via 192.168.1.1 dev eth0

  This works and is relatively straightforward, *but DHCP daemons and such like to undo what we've just did, unfortunately*.

And right below that, is the reason WireGuard doesn't use the OpenVPN technique:

  So, instead of replacing the default route, we can just override it with two more specific rules that add up in sum to the default, but match before the default:

  # ip route add 0.0.0.0/1 dev wg0
  # ip route add 128.0.0.0/1 dev wg0
  # ip route add 163.172.161.0/32 via 192.168.1.1 dev eth0

  This way, we don't clobber the default route. This also works quite well, *though, unfortunately when eth0 goes up and down, the explicit route for demo.wireguard.com will be forgotten, which is annoying*.

Both emphases mine.

Is there any current, Linux Administration book, or online resource, that would anyone here would recommend?

Most I found still use old and outdated commands, or don't really explain thing in depth.

It’s kindly explained on the actual WireGuard page as well:

https://www.wireguard.com/netns/

and it is mentioned on the blogpost as well. :)

I prefer flagging packets from cgroup. But since cgroupv2 it can only be done with systemd/slices, it's not easy and stable. (The slice must exist before iptables rule, and if it stops, i experiment kernel crashes etc.)

Someone need to come up with a new reference book on Linux routing. The last reliable book on the topic is about 20 years ago during Linux kernel 2.X, the time when the Linux kernel has a dedicate routing table as mentioned in the article. Now it is even more important due to the popularity of Linux networking in the era of cloud, edge and container.

[1] Book Review: Linux Routers - A Primer for Network Administrators, 2nd Ed

https://www.linuxjournal.com/article/6314

Vincent Bernat has one of my favorite blogs about networking.

https://vincent.bernat.ch/en/blog#tag-linux

I came to say pretty much this.

It's high time for somebody to write a guide to linux networking.

The closest thing to a comprehensive guide is the Linux Advanced Routing & Traffic Control HOWTO (https://lartc.org) but the last update (as far as I can tell from the website) is from 2012.

I'm not so sure people buy books like that anymore, not at the volume they did 20 years ago. When was the last time you bought an O'Reilly? I used to buy several of these a year, but can't remember the last time I even saw one.

You're right that a new, easy to read guide is needed, but I think it would have to be just a website.

The tools you need for Linux networking, mostly ip and tc, are as crude-but-powerful as they were 20 years ago, but now have much more functionality and complexity. Things are actively getting worse: a tool like brctl (bridge control) was obsoleted for the same functionality in iproute2 -- but brctl was much more intuitive to use.

Some nicer frontends than iproute2 would really help here.

I prefer the namespace approach with wg, as it becomes explicit and easier to use with services through JoinsNamespace. The routing inside the namespace is just using the main table.

This also allows to isolate the DNS queries by bind-mounting nsswitch.conf and resolv.conf as well as manage the isolated firewall for the namespace.

One thing worth adding to this is that what wg-quick does here - adding routes for each entry in AllowedIPs and a corresponding routing policy - is not part of wireguard itselfbut a wg-quick specific thing.

This can become important if you want to have redundant gateways or something like that - you can have overlapping AllowedIPs, it just won't work out of the box with wg-quick.

For example, systemd-networkd started also adding routes by default in v250 but (rightly so IMO[0]) reverted to not doing it in 250.3.

[0]: This comment captures it quite well: https://github.com/systemd/systemd/issues/14176#issuecomment...

Unrelated question. But may be someone knows enough. Is it possible to create high-available wireguard server? Like I put load balancer on UDP :51820 before servers and then those servers somehow share a session pool, so when one goes down, another can continue serving clients without interruptions.

Naive approach does not work: wireguard establishes connection and if backend server changes, this connection stuck for a few minutes until wireguard decided to establish new connection. Persistent Keep Alive does not help for some reason. Eventually it'll re-establish new session and will work from now on, but this delay is unacceptable.

Not a LB, look into Engarde.

Take a look a these (not tested these configurations personally):

- High Availability WireGuard on AWS [https://www.procustodibus.com/blog/2021/02/ha-wireguard-on-a...]

- High Availability WireGuard on Azure [https://www.procustodibus.com/blog/2021/06/ha-wireguard-on-a...]

- High Availability WireGuard Site to Site [https://www.procustodibus.com/blog/2021/10/ha-wireguard-site...]

Thanks, it seems that they just OK with user being offline for minute or two, they use ordinary configuration. I'm using the same approach, but I'd really love instant switchover.

"Fun fact: wg-quick uses the same numbers for the table and the fwmark: 0xca6c is just 51820 in hexadecimal."

An additional "fun fact" is that 51820 is also the default port number used by WireGuard servers.

Great article! Does anyone know of a similar resource for MacOS?

I’ve been trying to use Tailscale together with my company’s enterprisey CloudFlare VPN and Tailscale rules never seem to evaluate, even with split tunneling etc enabled. Need to dig into those routing tables one of these days…

Anybody working on an enterprise bundle and support package for wireguard? All "enterprise" vpn solutions I've had to touch in the past 10years are flaky at best.

Does tailscale match what you had in mind?

Has anyone been able to roll out a homemade speedify on Linux?

Reply on news.ycombinator.com