Pro:
+ Can just run "native" SSH directly over it (or, in our case, use x/crypto/ssh, without modification).
+ Lets `flyctl` offers a `flyctl proxy` command to users, so they can plug their own programs into whatever application they need to use, without asking us to change some proxy we run in our infrastructure.
+ Offers a single security and access control model (IPv6 private networking), rather than something we have to think about on a per-app basis.
+ In theory, we get all this right and never have to think about another network protocol in our infrastructure.
+ Allows existing network management tools and libraries to function directly with Fly.io infrastructure.
+ With the WebSockets gateway, we can do all of this stuff directly in browsers as well; that is, we can present TCP/IP as an API to browser Javascript to do UI stuff (and we're doing more and more UI stuff these days, in Elixir.)
+ Puts more IPv6 in the world.
+ Get to talk to Jason Donenfeld more.
+ Get to write blog posts like this.
Con:
- Spends one (maybe multiple) innovation tokens or whatever you want to call them.
- Way more things can go wrong; relies on state synchronization and on a clear network path between our users and our gateways. Right now, we have to care whether you can speak 51820/udp.
- User-mode TCP/IP via Netstack is probably significantly slower than a simple TCP proxy would be.
- Required `flyctl` to run a background agent process to manage multiple connections through WireGuard.
- The agent process adds to the list of things that can go wrong (hopefully we're ironed most of them out now).
I can probably come up with more cons.
...
It's buried in the middle of the post but I want to say it again because I think it's important: this sort of started out as a stunt; it's what I put together to allow people to SSH into their instances without having to install WireGuard locally, and that's all it was. I don't have to write a soul-searching pro/con list on stunts I use to give people SSH access, because lots of providers have super janky "pop a private terminal" setups. But all this stuff took on greater importance when we used it to run Docker for our remote builders.
I like the approach we're taking a lot! I don't... regret it? I don't think? I think I'm happy with it. But it's complicated.