BPF, XDP, Packet Filters and UDP
fly.io
fly.io
A few naïve questions:
> You can make any protocol work with a custom proxy. Take DNS: your edge servers listen for UDP packets, slap PROXY headers on them, relay the packets to worker servers, unwrap them, and deliver them to containers.
Curious: Wouldn't SOCKS5 here be a like-for-like replacement for PROXY? Why would one choose one over the other?
> WireGuard doesn't have link-layer headers, and XDP wants it to
Is the gist here that WireGuard doesn't because it is Layer 3? And that XDP sits one layer below it?
> Jason Donenfeld even wrote a patch, but the XDP developers were not enthused, just about the concept of XDP running on WireGuard at all
Could someone please explain this? Is it that XDP here didn't want to add a support to delegate routing onto WireGuard?
> It's a little hard to articulate how weird it is writing eBPF code. You're in a little wrestling match with the verifier
Would NetMap or Intel's dpdk instead make for an non-enterprising choice here? Don't they have a similar profile in terms of throughput? I guess, one has to use a userspace TCP/IP stack like gVisor's NetStack or LwIP to go with NetMap/dpdk?
> Those configurations are fed into distributed service discovery; our servers listen on changes and, when they occur, they update a routing map
How is this system implemented? Curious because uptime, availability, durability, and latency must be of prime importance for such a service. Is there a blog about this detailing the challenges inherent here? Or, does it use consul/etcd or some such out-of-the-box solution?
> a simple map of addresses to actions and next-hops; the Linux bpf(2) system call lets you update these maps on the fly.
Clarification: does this mean the maps are already in a format the bpf/2 command understands, or is something else going on here?
Thanks.
The nice thing about XDP is, you use llvm to make a .o, you load it with iproute2, and you forget about it; your code isn't a process running on the system, but rather a part of the Linux networking stack. It's as if the kernel just shipped with a "route Fly's UDP" feature.
I was going through Cloudflare's Magic Transit docs and they seem to simply use Direct Server Return (Facebook does so too for their L4 load balancer [0] instead of dealing with XDP in the return path).
Any reason fly.io doesn't but relays the packets again out via the edge?
[0] https://engineering.fb.com/open-source/open-sourcing-katran-...
Feels like Oprah Winfrey's September 13th, 2004 show: "YOU get a car! YOU get a car! And YOU get a car! Everybody gets a car!"[1]
>"You can make any protocol work with a custom proxy. Take DNS: your edge servers listen for UDP packets, slap PROXY headers on them, relay the packets to worker servers, unwrap them, and deliver them to containers. You can intercept all of UDP with AF_PACKET sockets, and write the last hop packet that way too to fake addresses out. And at first, that's how I implemented this for Fly."
This is really interesting. I looked at the linked blog post and was hoping there was more implementation details. Does your Fly pi-hole use HAProxy and the PROXY headers then? Is the config for that available anywhere i could see?
The first cut of this feature I built, without BPF, used NFQueue (diverting packets based on iptables rules to userspace), did a sockets-based proxy from edge to worker, and used a simple raw socket to fix the addresses and write the packet to its destination. NFQueue was annoying to work with, I looked at BPF filters instead, and ultimately wound up just doing the whole thing in BPF.
You don't need to know anything about this to use UDP on Fly.io; you can just add UDP ports the same way you'd add TCP ports (the `fly.toml` in the Pi-hole blog post shows an example).
The XDP project itself has a pretty excellent tutorial:
Anyone find this?
So, that is the way we will get to run C++ code in the Linux kernel. And, soon enough, in the BSDs.
It is hard to get a sense of how janky all this is, or how amazing it is that all this Rube Goldberg gimcrackery can be made to work the wonders it is seen to do all day, every day. It's not just a dancing bear, it's a bear on the Bolshoi stage!
(Donenfeld had better get his act together and get wireguard fitting better with how eBPF wants things to be, because that is where the world is headed.)
If your program isn't spending most of its time inside the kernel running code sorta JITted from eBPF, you're just not serious about performance.
Unless, of course, you have gone full-on kernel bypass, and the kernel never gets your packets at all. Then you can just run straight-up, optimized native machine code translated directly from C++, or Rust, or even, with masochism enough, C!
Can you clarify (and possibly tone down) your comment? The impression I get from the article is that he is doing what he can, even proposing patches.
As for Donenfeld: you couldn't be more wrong. Jason wrote a patch to fix the WireGuard/XDP breakage, and the XDP team rejected it, saying that they didn't feel XDP made sense for WireGuard.
Their position is also easy to understand: the point of XDP is to intercept packets before they're copied into socket kernel buffers, and you can't meaningfully do that with a virtual network that runs off UDP sockets to begin with. I disagree with them about this being dispositive --- consistency of interface is much more important to me than "performance surprises" --- but, whatever, at least acknowledge the debate rather than sniping.
People usually want HTTP + TLS handled for them, though. So when we ship QUIC + HTTP3 as a first class feature, we'll terminate QUIC and give people whatever their app process can accept.
We're not in a huge hurry to support QUIC / H3 given its current adoption. However, our users' apps will be able to support it once UDP is fully launched, if they want to.
Writing proxies is fun. Highly recommended.
We settled on Rust back when we were building a JS runtime into our proxy. It's a great language for hosting v8. When we realized our customers just wanted to run any ol' code, and not be stuck in a v8 isolate, we extracted the proxy bits and kept them around.
I think you could build our proxy just fine in Go. One really nice thing about Rust, though, is the Hyper HTTP library. It's _much_ more flexible than Go's built in HTTP package, which turns out to be really useful for our particular product.
I was actually just playing with Hyper for a few hours last night. Are you guys using async/await yet? Any suggestions for learning materials for async rust other than the standard stuff?
Ultimately I realized that most of what I needed from Caddy was really just certmagic, which has worked flawlessly since I got it set up. Plus I need the final product to compile into a single binary. Since my custom reverse proxy only took a few lines of code, I haven't worried too much about it. But there are a few features which I'll have to integrate eventually.
If I end up seeing myself headed down the path of making a full-fledged reverse proxy, I'll reconsider trying to implement my project as a Caddy plugin.
The tldr is: Docker containers don't offer enough isolation for multi tenant systems.
They're also very slow to boot, compared to a Firecracker VM.
But also, we run Firecrackers on our own physical servers and performance is quite good (even network + disk performance).
Like, if not even AWS, Cloudflare, and fly.io use containers, how can K8s be native in any way?
I mean, that makes even Lambda, which runs on Firecracker VMs, more native than K8s.
But I also think it's fair to call "Firecracker VMs" containers. Most of what those people are talking about is application packaging and deployment, not necessarily what actually runs.
For what it's worth, I am also cynical about "cloud native".
You can do a lot with TCP, and be tolerant to out-of-order delivery and drops, just by shuttling the individual packets. So we can in fact "cut through" TCP sessions directly to Firecracker, avoiding our proxies. We don't, though: our "tcp" handlers actually route through our Rust proxies, both because that's what they've always done, and because in most cases there isn't much of a win to bypassing the proxies, which have a lot more load balancing and resiliency logic than the BPF-based UDP data path does.
We don't use any of Nomad's networking stuff, really, so all the BPF work was relatively easy to bolt in. All our Firecrackers get tun/tap pairs we manage outside of Nomad, it's reasonably simple to make them do what we want.
I think we'll ultimately move off for custom orchestration but we want to wait as long as possible before we do that.