WireGuard Bounce Server Setup
gitlab.com
gitlab.com
If you experience this when logging in to a host via SSH, the delays are almost certainly due to either missing or non-functional forward/reverse DNS (the SSH server will perform these lookups when connections are received).
You've got at least three options (in order of preference):
- Add the appropriate (A/PTR) resource records to DNS so that the queries get a valid response
- Add entries to /etc/hosts file
- Set "UseDNS no" in the /etc/ssh/sshd_config file and add "-u 0" to the parameters passed to `sshd` when it's started (how to do this will vary by distribution/operating system)
Dec 15 22:23:14 ip-99-99-99-100 sshd[1995]: pam_unix(sshd:session): session opened for user admin by (uid=0)
Dec 15 22:23:14 ip-99-99-99-100 systemd[1]: Created slice User Slice of UID 1000.
Dec 15 22:23:14 ip-99-99-99-100 systemd[1]: Starting User Runtime Directory /run/user/1000...
Dec 15 22:23:14 ip-99-99-99-100 systemd-logind[510]: New session 34 of user admin.
Dec 15 22:23:14 ip-99-99-99-100 systemd[1]: Finished User Runtime Directory /run/user/1000.
Dec 15 22:23:14 ip-99-99-99-100 systemd[1]: Starting User Manager for UID 1000...
Dec 15 22:23:14 ip-99-99-99-100 systemd[2001]: pam_unix(systemd-user:session): session opened for user admin by (uid=0)
Dec 15 22:24:44 ip-99-99-99-100 systemd[2003]: pam_unix(systemd-user:session): session closed for user admin
Dec 15 22:24:44 ip-99-99-99-100 systemd[1]: user@1000.service: Main process exited, code=exited, status=1/FAILURE
Dec 15 22:24:44 ip-99-99-99-100 systemd[1]: user@1000.service: Killing process 2007 (gpgconf) with signal SIGKILL.
Dec 15 22:24:44 ip-99-99-99-100 systemd[1]: user@1000.service: Killing process 2008 (awk) with signal SIGKILL.
Dec 15 22:24:44 ip-99-99-99-100 systemd[1]: user@1000.service: Killing process 2013 (dirmngr) with signal SIGKILL.
Dec 15 22:24:44 ip-99-99-99-100 systemd[1]: user@1000.service: Failed with result 'exit-code'.
Dec 15 22:24:44 ip-99-99-99-100 systemd[1]: Failed to start User Manager for UID 1000.
Dec 15 22:24:44 ip-99-99-99-100 systemd[1]: Started Session 34 of user admin.
https://serverfault.com/questions/954844/chroot-gpg-agent-an...
In this case, GNUPG is trying and failing to start (likely misconfigured), this in turn means system failed to start the likely only user service configured, hence it fails the session setup. The session is still created (managed via logind) to provide a shell and PAM environment.
On a non-systemd this stuff would be managed by other scripts and tools that do essentially the same thing (minus being able to start user services).
It also creates a slice for the session, which enables systemd to kill all processes created in a session when the session ends (logout), minus some exceptions that can manage themselves (like Tmux). Helps against people who forget that they backgrounded and nohup'd a script (which isn't a good way to do things that are supposed to keep running).
If you want to see what services are available in your user session run 'systemctl --user'.
In this case, gnupg was, both for that user and for that host, wholly unconfigured -- evidently it was just roped in by some package dependency. So, trying to run something that depended on the system or user having gnupg configured correctly (or at all) was wholly wrong.
Systemd killing my nohups does not seem like doing me a favor. If I had wanted the process killed when my connection dropped, I would not have nohup'd it. (Later in the article, I use nohup as intended.) How do I turn that off, without breaking other things? Or, failing that, what is the right way to get that behavior, without nohup?
And, what controls which services the session manager will try to run on behalf of a user who has not tried to customize anything? It would probably have been better to turn pushy gpg-agent off, in some sysyem-wide way, than to have entirely deleted it.
You can run 'loginctl enable-linger' to disable this behaviour or set 'KillUserProcesses=no' in /etc/systemd/logind.conf
You can also use 'systemd-run --scope --user /usr/bin/yourscript.sh', this will explicitly keep the process running even when the session closes, even if lingering processes are killed. It'll also keep track of all processes created by that script, so if the script exits, it'll kill all processes that are left over and cleanup. It also meshes with other parts of systemd so `systemd-run --scope --user --on-calendar=daily /usr/bin/script' will run the script daily until you tell it to stop or the machine restarts (since it's not configured on disk). You can even keep the service if the script dies with '--remain-after-exit', which tells systemd to keep the service alive as long as processes for it exist or you terminate it manually.
If whatever your running in the background needs forking, you can create a service and start it under your user, that properly keeps it up and running.
Systemd understands a nohup as "don't terminate the process when the terminal closes", not as "don't terminate the process when the session ends" which can be different events depending on the setup (ie, graphical terminal or tmux).
The session manager will start any user services that are activated, your distro usually configures that in '/usr/lib/systemd/user'. Manually activated units are in '/etc/systemd/user' if they are system-wide and user config is in '$XDG_CONFIG_HOME/systemd/user' or '$HOME/.local/share/systemd/user'.
To turn it off you can deactivate it via 'systemctl --user disable gpg-agent' for the current user or 'systemctl --global disable gpg-agent' for all users. If it activates via socket activation you can either disable the socket 'systemctl --global disable gpg-agent.socket' or mask the service 'systemctl --global mask gpg-agent'. Masking the service prevents it from being started for any reason.
https://github.com/anderspitman/awesome-tunneling
Always on the lookup for tools I may have missed.
You end up with another "zt" interface, so you can apply firewall rules to that interface, separate from your main NIC(s), kinda like assigning your own VPC but for your laptop, etc. It only takes a few seconds to install the agent and join your private network, and it seems to reconnect quite nicely as well; if your network is private, you can force all new attempts to connect to be approved (where you can check the mac address first). Basically, it's exactly the right level of technical config traits with a simple interface.
(no affiliation, just a very happy user, but I will say that running your own seems to be a lot more difficult than it should be..)
0: https://github.com/zerotier/ZeroTierOne/blob/master/LICENSE....
As Frank Robinson [0] once said:
> "[...] Close only counts in horseshoes [1] and hand grenades."
--
[0]: https://en.wikipedia.org/wiki/Frank_Robinson
[1]: https://en.wikipedia.org/wiki/Horseshoes (for the uninitiated)
http://www.basicinstructions.net/basic-instructions/2019/8/1...
We did an audit of it and they fixed lots of configuration problems, it's now pretty solid security defaults wise. And the WG integration works well.
They support the following setups: OpenSSH Tinyproxy OpenConnect / Cisco AnyConnect Stunnel Shadowsocks Obfsproxy WireGuard,
Running on: Amazon Web Services (AWS) Microsoft Azure Digital Ocean Google Compute Engine (GCE) Linode Rackspace
Anyone have anything like an introduction server to help wg peers behind nat find each other?
Not an intro or tutorial, but I've not found a better write-up than on their blog:
https://tailscale.com/blog/how-nat-traversal-works/
(personally, I'd just do what the author did... it's a ton less work to setup and maintain)
I'm still looking for a FLOSS mesh network built on top of wireguard that can do NAT traversal between nodes and fall back to tunnelling traffic for the annoying cases where this fails. I don't really want to use Tailscale because (1) their server is not open (though this should not matter a ton for trust) and (2) they require a Google account or similar to sign up.
1: https://www.jordanwhited.com/posts/wireguard-endpoint-discov...
Alice finds Bob's external IP:port using the registry. That makes sense. But doesn't Bob need to send a packet to Alice to setup the NAT traversal on his side? More accurately, Alice uses the SRV field to populate the wg peer information on her side -- but how does Bob know that he needs to update the peer information on his side?
I think this is really close, but maybe using a custom DNS server for this might be trying to be a little too clever?
(1) Alice and Bob both connect to coordination server
(2) Alice sets Bob's endpoint to bob_ip:bob_port
(3) Bob sets Alice's endpoint to alice_ip:alice_port
(4) Both try to ping each other, which makes both of them originate outbound packets from the wireguard socket.
IF Alice and Bob have the same public IP:port pair in their NAT with each other as they do with the coordination server (which turns out to be true in tests on my EdgeRouter X NAT, but certainly isn't true 100% of the time), then I believe this process will result in a working connection.
If you're asking how Bob & Alice know that the information in the coordination server has changed and that they need to reconfig their endpoints, then I'm not sure. Perhaps they could just re-query the coordination server on a timeout or whenever their connection went down.
I agree - using DNS as a transport seems orthogonal to the NAT punching process, and maybe too clever. If you're going to need to write custom endpoint code either way, I don't see a huge advantage. If wireguard supported DNS-based endpoint information by default (automatically re-resolving IP and detecting port from SRV), then it'd make a lot more sense.
It assumes full-cone NAT - you punch a hole once by sending a packet to a third party, like a stun server which will along the way tell you your external ip:port, and then a full-cone NAT would forward packets coming to that ip:port back to you, regardless of the source.
If it can be, you can just have the NATed peers connect to it, with persistent keep alive set so they maintain it.
echo 1 > /proc/sys/net/ipv6/conf/all/forwarding
echo 1 > /proc/sys/net/ipv6/conf/all/proxy_ndp
for i in `seq 0x0010 0x001f`; do
ip=2001:1111:2222:3333::$(printf '%x' $i)
ip neigh add proxy $ip dev ens3
done
ip addr add 2001:1111:2222:3333::10/124 dev wg0
Um, no?
It clearly works. It's worked for a long time. But the common opinion still seems to be "well, this isn't supposed to work, so using it is a bit dodgy..."
Traversing non-cooperative fully symmetric NATs, which randomize ports, is hard enough also for pwnat. Though in theory should be doable - you just need a lot of patience to brute force ports (there's only 64k of them) until it finally clicks
"Conclusion: ... the presented method works ... virtually never if both peers are behind NAT."
Still, "virtually never" is not "never".
ssh laptop.wg.mydomain.net
which resolves into 10.44.0.3 for example.
There are two major downsides with this approach:
(1) all of your traffic must go by way of the bounce server, even if your machines are on the same LAN.
(2) the bounce host can observe all of your traffic.
I've been considering adding a second layer of wireguard on top of this for end-to-end encryption between nodes. Each machine would have one interface configured like this article suggests simply to assign a stable NAT-traversing IP address. Then they'd each have a second interface using the first-level IP addresses as endpoints. This would result in 2x the encryption overhead, both in CPU and more importantly in MTU, but it would mean the bounce host was no longer able to intercept even unencrypted data. If you run a daemon on all of the hosts, they can play with the endpoints used on this second level network to avoid using the bounce host when they find they can directly connect to each other, but the default path would work (if somewhat slowly) by the double tunnel process even without this daemon.
For bonus points, run multiple bounce hosts, connect to all of them from each peer, then select the endpoint to use for each paired peer connection based on the total path latency.
--
[0]: https://en.wikipedia.org/wiki/Dynamic_Multipoint_Virtual_Pri...
A [your choice of HW] would do fine, if it runs only your code."
Idea: "Colocation centres" for users' computers instead of data centres for users' data. As directed by the author here, users would store no data on these computers.1
1. By their nature, each user-owned supernode computer would provide some discoverable metadata, as would any router, namely, the IP addresses of the members of the private network/s it supports and which of those IPs are connecting to each other, but this would be unlike the centralised repositories of millions of users' metadata we have now, managed by private tech companies.
As for the rest of this writeup, it sounds much like the need for a supernode in "LAN-over-Internet"-type P2P networks that encapsulate Ethernet-like packets in UDP packets. The supernode does not necesarily need to route traffic between nodes behind NAT (rarely necessary), it only needs to store the equivalent of an ARP table accessible by all of them.
You mean this? https://en.wikipedia.org/wiki/Colocation_centre
This is basically AWS Workspaces. It can work, but it's annoying, expensive, and various latency and bandwidth issues are a killer.
1: https://www.jordanwhited.com/posts/wireguard-endpoint-discov...
What did the author mean by this? I can understand amazon employees having the ability to peek at private data but what about "hackers"? is he referring to the fact that lightsail is containerized and not running on VMs?(just a guess)
If that does what you need, then it is a reasonable alternative. I would prefer the shorter routing you get. But if one of your NATs is corporate, a phone carrier, it might not work.
If you might want to provide other services, particularly a remote exit node or web service, the bounce node is a good start on that.
That's what makes this method simple... with the downside that you need a publicly exposed intermediary. But it is vastly simpler to setup than STUN.
For minimal setup, zerotier looks like the winner.
Same thing for a router configuration to accept inbound wireguard requests on a home network with dyndns. You set it up once and are done.
I see the benefit of the bounce server if you operate a network in an environment where you don't have the ability to control the router config; however, when you do have the ability to update firewall/router config, then I'd prefer just setting up a domain name and avoid the dependency on a third party server.
But you do not seem to be getting that the use of nftables, for the open-network bounce server, is wholly optional.
I am not sure the nftables configuration I have is right... It might permit using my bounce server to forward packets that then appear to come from it, if they happen to mention the right port. I would welcome advice.
After further investigation, I have discovered that dyndns would not solve my problem, because the firewall at one end is especially picky; even zerotier and tailscale admit (grudgingly) that they use bounce servers for such clients.
This isn't as easy as A <=> B, but it is probably a common enough use-case. It was really hard to figure out what I needed to put in my wg.conf files. But once I got it, it was really solid.
Is there something like "WireGuard in anger" or even something like the GitHub introduction to Git for WireGuard? It seems like it would be very helpful.
Internet -> [eth0] WG BOX [wg0] -> [wg0] HOME SERVER
analogous to:
Internet -> [eth0] ROUTER [eth1] -> [eth0] HOME SERVER
Also you probably want to setup some sort of NAT instead of just routing packets directly.
https://launchpad.net/bugs/1890201
https://launchpad.net/ubuntu/+source/wireguard/1.0.20200513-...