JIT WireGuard
fly.io
fly.io
I don't seem to understand. You can add peers at runtime, e.g.
https://serverfault.com/questions/1101002/wireguard-client-a...
Can somebody clarify?
EDIT: If I understand correctly, that step is already too late. They want to authenticate a peer before adding it to the interface in order to prevent stale entries on the interface.
They thus put a eBPF filter in front of the interface and do the cryptokey-routing based association to an authorized counterpart by themselves. If it checks out, then they add the peer to the interface and remove it after a timeout.
If you're a VPN provider (for instance), the current API is a little clunky. It's not just that at any given time only a small fraction of your peers are actually in use, though that is probably true. It's also that as the number of peers you handle scales, from hundreds of thousands to tens of millions, you lose the ability to store them all in a single instance of the kernel at all; there are just too many. If peers have to be pre-installed, a consequence of that is that they get locked to specific server machines.
But, as the post points out, you can get a facsimile of the interface you need today with simple packet capture. And Jason set the API up so that you can --- very easily --- flip the initiation from server to client so the connection experience is seamless, even though the kernel dropped the first initiation message (because the peer hadn't been installed).
So that's the idea here.
Jann Horn pointed out that we could have taken this a step farther: we could have held on to the initiation packet that we captured, and, once the peer was installed, replayed the packet into the kernel. Which is also a neat idea.
I don't think there's much in this post that is going to change your life. It's just a couple of neat tricks that we thought people would like to know about.
(Though: the next step for us is to build on this to get "floating peers", de-regionalizing them completely, so users no longer have to think about what region their peers are configured in, which I think will actually have a product benefit for users, unlike this, which has primarily nerd benefit.)
This doesn't implement the completion (replay or reverse initiate) which I think both are also novel approaches to this.
So exciting!
JIT Wireguard is a weird way to frame this. My mind went to “why? The performance bottleneck is the crypto and per client JIT won’t help with that.”
I would have just gone user space. Use something like tokio-uring or glommio to get the performance. If you keep going in the kernel you are going to keep hitting limitations because Linux is not built to serve millions and millions of active tunnels. Even doing millions of TCP connections per kernel gets hairy sometimes.
Every limitation will require a hack. Every hack will be some system config that has to be applied and managed. The tool chains for provisioning Linux metal boxes are vastly inferior to the tooling for developing apps and services and managing their config.
Or am I stupid and misunderstanding?
And JIT just as in "just in time" configuration of Wire guard. Once the configuration has been done, their stack stays out of it.
I think JIT compilation wasn't popular in ancient times, so I never associated JIT with compilation by default.
Based off the excellent work in done by wireguard-go but I've attempted to simplify and make things a lot more idiomatic for library use.
I reckon building a service mesh out of it would be interesting, obviously supporting multiple languages would be hard but maybe you could implement a sockets API. Though it might be hard to compete performance wise with mTLS as I've not seen HW acceleration for WireGuards crypto yet.
FWIW, I'm currently on the market for freelancing roles, so if you're interested in Golang freelancers in the high-speed/secure networking space, please reach out (email is in my profile).
I have a dream of taking a userspace Wireguard project like this and gluing PAKE over a relay in front of it to exchange Wireguard keys, followed by holepunching and establishing a direct tunnel. Basically Magic Wormhole for arbitrary tunnels - and hopefully vastly improved performance for the file transfer case as well, rather than crapping out at 20-30 MB/s over long fat networks.
I do a lot of work on remote servers for heavy data processing (large files). Sometimes I need to get these files back to my local computer, or just view figures I’ve generated using remote data. I have a small reverse tunnel daemon setup where I can (in my remote terminal) send arbitrary files or data back to my local computer.
$ rtun send file.txt
$ rtun view data.pdf
However, this setup is a bit cumbersome and I can’t just run `ssh host`. If I could have the same setup with an in-app wireguard tunnel, with no other setup, it would be amazing.
https://github.com/anderspitman/awesome-tunneling
Your use case sounds interesting and there may be a tool out there that will do it, but I can't quite wrap my head around your description of how everything is connected and what runs where with your current setup.
I agree with sibling that my main question is what prevents you from using SSHFS or similar?
I've been looking for interesting applications for noisysockets, and I'm obsessed with network attached storage so this is a fun one for me.
But really, it think it's that I'm already in a terminal connected to a remote system. I don't want to have to go to a different terminal to try and transfer data that I'm already looking at. And trying to use a Finder window (or explorer) to navigate a complex remote filesystem hierarchy isn't fun.
Occasionally I can do my work locally, but usually the data is large enough that I have to do my work on a remote server/cluster. When I generate figures describing my data, I want to see those locally. This particular use-case could be solved by using something like Xpdf, but it's easier to send the figure back to my local machine and view it with Preview.app. If I could view a PDF figure in my terminal -- I'd probably just do that.
I also sometimes do need to send datafiles back to my local computer. In these cases, I could use sshfs (but don't like the duelling terminals) or scp (but my file paths can be long and complicated, so typing out paths is a pain). I used to actually just handle this with Dropbox. I'd have a program that would send files to a specific Dropbox folder and that would then sync to my local computer. That worked well, but the delay between syncing was an issue. I've also tried using a remote tunnel back to an SSH server running on my laptop. That was equally cumbersome.
The two other limits I would like to avoid: 1) no ports, unix sockets only, and 2) no coordinating servers. I try to use only unix sockets so that I can use normal unix permissions to protect the socket. I'd rather not also include an authentication layer if I can avoid it. I'd also like to not need a third-party server to coordinate the connection between the server and my local computer. My remote servers are behind VPNs and while they can access the internet, it's heavily firewalled. If I already have an SSH connection to the remote server, I'd like to tunnel through this if at all possible.
Here's the code/project I wrote to manage this: https://github.com/mbreese/rtun
I actually had some folks from OpenZiti reach out and they have another similar tool in this space.
Wouldn't a lost message just mean NATS would retry delivery until succesful? Anyone know why they would experience noticeable unreliability?
The one thing where I can see an impedance mismatch is ephemeral single-message connections which I don’t think it’s built for. In either case, more details would be invaluable. (Fly folks, please consider sharing!)
I believe if you're using core nats (not JetStream) there's no option for re-delivering like at all.
Is this(effectively) adding a half round-trip to the handshake? i.e.
1. ->flyctl sends Initiation
2. <-peer is added via netlink(which causes new Initiation to be sent)
3. ->Response from flyctl 1. -> Alice calls Bob
1.a. Bob does not pick up the call, but adds the number shown from caller ID to his address book
2. <- Bob calls the number (Alice) back
3. -> Alice picks up and they talk happily“We’ve gone a step beyond that: every time you run flyctl, our lovable, sprawling CLI, it conjures a TCP/IP stack out of thin air, with its own IPv6 address, and speaks directly to Fly Machines running on our networks.”
The reason to mention conjuring a TCP IP stack out of thin air, is that most of the time the operating system is providing the tcpip stack as part of the kernel. With wireguard go, the tcpip stack is running in user space, meaning it can be created in a normal user space process such as the flyctl command line interface. This is indeed quite magical for people who have been around for many years as a usable in-process userspace tcpip stack is a relatively new and a novel thing.
[1]: https://retrocomputing.stackexchange.com/questions/5891/how-...
You can’t make connections with other processes over the same connection flyctl makes when it’s doing stuff.
That is what GP is saying.
I'm not sure if initial handshakes can be replayed, but since WireGuard ignores unknown clients, it might be possible.
Out of that, you can construct a userspace process that reads a packet from the queue, decrypts the noise initiation, requests configuration, adds wireguard config via netlink, and then releases the packet. And you can do that with multiple packets in flight concurrently.
It also allows things like fail-open if the userspace process is broken or overwhelmed, which would be useful here (already in-kernel wg peers would keep working), and spreading load over multiple workers.
https://netfilter.org/projects/nftables/manpage.html (search for queue)
https://wiki.nftables.org/wiki-nftables/index.php/Queueing_t...
Here's a nice & simple Rust library for the userspace part, to give an idea of what the shape of the API is: https://docs.rs/nfq/latest/nfq/
Also, I'm available for contract work ;) Say hi to Ben from Tv.
(I say this ruefully, having lost the argument about what our default should be.)
But the company feels a little immature. Once our API server became unreachable in Fly for 48 hours. I'm not sure if it was my fault for getting config wrong or if they just had another "silent" failure. They have a "db" product, but it's "not managed postgres". Would get consistent disconnections from that. Just feels weird for them to add a top level noun in their cli for postgres and then limit the extent it's a feature they support.
API access to their core service would frequently go down and leave us waiting to deploy new service fixes.
I miss the deployment experience, but I'm frankly happier with Cloud Run on GCP. Just way fewer "surprises" and much more complete documentation.
It's wild to me how little support VPS providers have for reducing latency of backend services for users. None of them support Anycast, and there are very few GeoDNS options (it adds more complexity besides).
I just wish Fly.io had cheaper data transfer, since I'm currently having to re-implement (poorly) a lot of their features for an ngrok-like service I'm working on.
[0]: using them for https://lastlogin.io
[1]: Here's all the fly-specific code necessary to run LastLogin in a globally distributed way: https://github.com/lastlogin-io/obligator/blob/37f75cc861f1b...
Why would anyone choose a provider that is inferior (apart from toying)?
I mainly read two kind of stories from fly.io. Their promotional, but well written and interesting technical blogs like this one and stories about issues with their services and miscommunication. So, despite liking their blogs I don't consider using it.
That said, I've had very few issues with their platform, and I don't think it's ever caused downtime for my (admittedly very small) service.
Not affiliated, just a satisfied guy that needs access to private AWS VPCs across multiple accounts and would love to see them be more widely adopted.
Tell me again why p2p privacy software needs to phone home?
My goal is to access private resources via SSH bastion/jump machines in a specific account. There's a few ways to do this in AWS, but all of them are more costly by a pretty wide margin.
aws ssm start-session --target $instance-idTheir site is extremely vague.
My thoughts exactly as I was reading the first paragraphs.
> Note that there’s no API call to subscribe for “incoming connection attempt” events. That’s OK! We can just make our own events. WireGuard connection requests are packets, and they’re easily identifiable, so we can efficiently snatch them with a BPF filter and a packet socket.
Nice idea.
> When we get an incoming initiation message, we have the 4-tuple address of the desired connection, including the ephemeral source port flyctl is using. We can install the peer as if we’re the initiator, and flyctl is the responder
And this works behind NAT?
Sure, UDP NAT only knows the 4-tuple (say {wggwd.fly.io, 12345, clientIP, 23456}).
Any UDP packet, whether it's a new "initiator" udp packet, or a response to the outgoing initiation message, will look the exact same to any UDP NAT in the way since it only has the 4-tuple to go on, and the 4-tuple is the same.
I doubt that happens much, and I assume you'll fall back hapilly enough to the second time the "client" sends an init packet and then you simply respond with the response packet.
cd directory/with/Dockerfile
flyctl launchPlus, you could have wireguard files on your local or remote filesystems, or any character device, or named pipes if you felt like it. You could use a "jit" daemon to build tap or other interfaces for you, or just do it individually at the application layer. You could have pre registered keys with the kernel, or you could manage that directly, or generate them randomly.
It's always been a weird smelling underspecified IPSEC clone to me, when it could have been so much more.
Why? WireGuard is a VPN, it's pretty normal for VPN solutions to expose themselves as a network interface.
> It really should have been a generic "filter" that you could attach to any type of file handle.
What's the use-case you had in mind here? I'm not sure how generifying it to a "filter" on any type of file descriptor looks for an interactive protocol like wireguard.
> It's always been a weird smelling underspecified IPSEC clone to me[...]
Just because there isn't an RFC? I've always found the wireguard paper[0] to be quite readable and thorough in it's specification of the protocol.
Though there are some nuances with the routing selection/filtering too, which gets troublesome when you just need a pipe and run a proper routing protocols over it. ::/0 solves most of it but still there are some rough edges.
[0] well, for *me*
[1] One of the amusing things I discovered what I have a full 10MB/s+ to the SMB server in the DC over the WireGuard tunnel (and that's because it's 100Mbit/s uplink), while the Synology which sits on the same router on a 1Gbit port only makes 3-5MB/s.
My issue with wireguard is that it's not enough for full-fledged VPN solution, for example there's no way to push routes to the client or DNS configuration or something like that. Those are very basic needs. If you have 100 users and you decide to change routing scheme, well, you're in a trouble. It is supposed to be solved by higher-level protocols, but I'm not aware of any open standard-de-facto ones with quality implementations.
Because you can definitively configure routes and DNS at connection time.
Obviously there are many things you could do to allow this (run a routing protocol, build a custom client which gets route information, etc), but the "out of the box" wireguard is a kernel interface, a wg command, and a utility script (wg-quick). I think there are some gui based clients for non-linux based OSes, but it's the same principle.
DNS is nothing to do with the wireguard kernel or userspace, it's configured in the "wg-quick script" (there's a bash function called set_dns), but you can do that however you want.
Wireguard alone isn't what an enterprise would consider to be a "VPN solution", it doesn't push configs from a central location, it's very much a peer-to-peer tool. You can build "enterprise" features like centrally defined routes or DNS on top of that, or not, it's not opinionated.
(* for linux and bsd)
though I don't know what would prevent any bgp daemon from running in e.g. the wireguard iOS app? there are bgp daemons like gobgp that can be easily integrated in other software.
but this was more meant to be a joke than anything. wireguard is batteries definitely not included, and is why tailscale and the like do exist.
Yep, this one too.
The cryptokey routing is pretty fundamental to wireguard, I'm not sure you could have one without the other.
In any case the point is I would prefer to just have the basic components available and let me piece them together however I want. Mostly to allow using the underlying technology in more contexts that it is currently available in.
In software you often choose between a small monolith and a big kitchen sink. Once you have 1 more need than the monolith covers, you have to go over to the kitchen sink.
In any case, really cool write-up! I wonder if they thought about making `flyctl` do a check with their API for any command that requires talking over wireguard to ensure the keys would be installed in the gateway. Since `flyctl` knows when the last command was run with it, it could do this only after some inactivity. And on the gateway machines, they'd just clean up any inactive peers with a cron (which they seem to be doing already).
Not a solution as elegant as the one they reached (which is super cool), but I'm assuming the considerably lower effort would make it appealing.
[0]: https://labs.leaningtech.com/blog/webvm-virtual-machine-with...
We choose to use Tailscale since they allow WebSocket-based connections via their DERPs.
It is interesting that, originally, DERPs were intended to be a solution for machines in extremely limited networking environment where nothing but HTTP is allowed. Turns out browsers are exactly one of those extremely limited networking environments.
Some context that I wasn't initially aware of: apignotti is the CTO of Leaning Technologies, which is where the article GP linked is from.
MSFT RDP (video):https://youtu.be/1NMrxRIowog
Private network for Grafana (video):https://youtu.be/l5ktiI-j3eg
Private network for Plex (blog post)https://blog.openziti.io/its-a-zitiful-life
Basically you decide what 'app' you want to deliver via the overlay, e.g. Grafana, Plex, RDP. For those destinations, a (one time) bootstrapping process (invisible to end user) results in your browser receiving a <script> tag which includes some configuration when the browser attempts to connect to the destination (Grafana etc). This ultimately results in the browser downloading some JavaScript and WA, and registering a service worker (the wasm contains the PKI bits).
After successful auth, your browser can then open a websocket to your private OpenZiti overlay network (distributed, OpenZiti overlay network software routers, deployed where you want them), and ultimately hit the app (which no longer needs to listen to anything other than the overlay network; becomes private).
Desktop Chrome is the most tested, followed by Android Chrome.
The way we think about things, if we were going to try to provide a browser experience of doing something with WireGuard, we'd probably just fork off a tiny Fly Machine VM to run it on. Just a different vibe here.
This means that WireGuard-the-org can't distribute the mac GUI app directly, and if you want to use the official WireGuard app (eg for accessing a VPN for privacy), somewhat ironically you have to ID yourself (which requires a phone number) and your computer's hardware serial to Apple first to download it via the App Store. It smacks of the totalitarian rms "right to read" essay - provide strong ID and hardware unique identifiers to be allowed to download and use privacy software on your own computer.
https://www.wireguard.com/install/
While not directly relevant to fly.io, I figured this might be relevant in the context of Apple's other anticompetitive actions this week related to web-based native app distribution on iOS in the EU for the DMA. This issue with VPN apps has been true on macOS for years.
I personally didn't know why WireGuard didn't offer direct downloads, but I emailed Jason Donenfeld a couple years ago and he let me know that Apple has been restricting these APIs in non-AppStore apps, which was news to me, as I didn't know that Apple had started any of their AppStore-only bullshit on macOS.
It appears that they will now give out these entitlements for non-MAS apps, but only to members of their developer program.
This means that you can't build one that will work on macOS without IDing yourself to Apple, so you still can't build it from scratch from source (as a non-developer-program person) and expect it to work (without disabling SIP).
Note defending the current state of things, but I can understand how we got here and it may not be malicious on Apple's part.
That's bullshit and I'm pretty sure you know it. This is a technical forum and you should know better than to spread unsubstantiated Apple-bashing FUD.
MullvadVPN and ProtonVPN are two very well known VPN providers who's VPN apps you will not find on the Apple App Store and instead you are invited to download from their respective websites. It has been that way basically forever (or at least a very long time now we're in 2024), so its not like its a reflection of a recent Apple change either.
I actually quite like the fact that WireGuard is distributed via the Apple App Store and I thank them for doing that. It makes updating much easier, rather than having a dozen apps each with their own update process.
I encourage you to try to build a working copy of the official WireGuard macOS app from source without an ADP membership. You can't.
https://git.zx2c4.com/wireguard-apple/about/
https://git.zx2c4.com/wireguard-apple/tree/Sources/WireGuard...
Your original statement is still bullshit.
You said "Apple simply will not provide the required VPN entitlements for web/self distributed apps".
I've clearly demonstrated that statement is just plain wrong. MullvadVPN and ProtonVPN are not distributed via the AppStore and they work and are not blocked by MacOS.
then correct your initial post, which says something entirely different:
> Fun fact: the WireGuard macOS client application cannot work on macOS unless it's distributed via the App Store - Apple simply will not provide the required VPN entitlements for web/self distributed apps. You can use the commandline wg tools (which use a different OS API) but not the GUI ones.
it really is increasingly annoying that people just post nonsense on HN to back up their dumb prejudices.