Ooooh I love Wireguard, we'll have an article about this in the next couple of months.

We picked it because it's really simple to manage, and we wanted to ensure traffic between datacenters was always encrypted. We have a little tool called "flywire" that keeps wireguard peer configs updated from Consul. Once we accept a connection from a user, we pick a target VM, and then connect them over the wireguard mesh.

For our purposes, it basically doesn't add any noticeable latency. I think when we tested we say something on the order of 0.1ms of added latency over wireguard, but I don't quite remember. It's never been the source of latency problems when we do have them, at least!