IPv6 WireGuard Peering
fly.io
fly.io
tons of PaaS providers are no help at all if you want to run an app that uses ZeroMQ or basically anything that isn't http.
One super small nitpick though, and maybe it's just me, but
> Technically, what we end up delegating to each instance is a /112, which is the IPv6 equivalent of an IPv4 Class B address
> WireGuard peers get /120 delegations (the equivalent of an IPv4 class C)
Should be
> Technically, what we end up delegating to each instance is a /112, which is the IPv6 equivalent of an IPv4 /16 network.
> WireGuard peers get /120 delegations (the equivalent of an IPv4 /24)
classes haven't really been a thing for almost 30 years.
If there is something with a socket of any sort that you can't do on Fly, I'd like to know about it, so I can make Kurt write some BPF code for a change.
If I were to run a container that handled tcp connections, is there any sort of max connection duration?
One of the projects I have worked on involved accepting tcp connections from cellular connected race timing boxes, dealing with their protocol and sending the resulting data into another system.
I guess you could think along the lines of syslog or MQTT.
This isn't very bandwidth or CPU intensive, but does require keeping the sockets open for hours at a time.
heroku/app engine/lambda/cloud run all replace CGI in one way or another, but none of them can replace inetd.
Then you'll never use "Class A/B/C" again. :) It's correct to just say /24 while Class C isn't, because it has a more specific but outdated meaning.
Soo...Happy 50th birthday? (all the 49.9999 year olds will get this joke...)
> You can configure an application to listen for global traffic on ports 80, 443, 5000, and ports 10000 - 10100.
Technically, when fly.io first launched, its usecase was fast, anycast "middleware" for web apps: https://arstechnica.com/information-technology/2017/04/pushi... :)
FOne interesting problem we have is that apps span regions, so the less information we have to propagate when VMs come up, the better. The 6PN routing is already established. When a VM boots it's available immediately because there's nothing to propagate.
Curious about fly's other design principles: A blog post about that would be wunderbar
(There's no possibility in our system of routing customer traffic outside of WireGuard, because our entire connectivity fabric is WireGuard; nothing but WireGuard, for instance, would know what to do with an fdaa address, or, for that matter, how to reach an instance's IPv4 addresses.)
We can filter and drop packets at ingress/egress with this design as well.
What's the advantage you see to having assignments out of per-customer /64s?
The workflow is literally: tptacek writes a post, we fix inevitable typos, and then we merge it.
Is it not feasible to make wireguard more flexible with a bit mask? The annoying thing with having to use 1:1 NAT in a packet pipeline (which is what you’ve done here) is that logs, metrics, etc get all screwed up for correlation because wireguard doesn’t see the same things everyone else does.
I guess putting wireguard in the kernel wasn’t such a hot idea after all?
Think about how big the BPF program is that implements the switcheroo we do to get around this. It's not, like, a big ask.
Are you using eBPF for this egress filtering? My (hopefully mis)understanding is XDP programmes, the higher performance cousin of eBPF, can only work on ingress packets currently.
> The unlucky bit is WireGuard. XDP doesn't really work on WireGuard; it only pretends to (with the "xdpgeneric" interface that runs in the TCP/IP stack, after socket buffers are allocated). Among the problems: WireGuard doesn't have link-layer headers, and XDP wants it to; the discrepancy jams up the socket code if you try to pass a packet with XDP_OK. We janked our way around this with XDP_REDIRECT, and Jason Donenfeld even wrote a patch, but the XDP developers were not enthused, just about the concept of XDP running on WireGuard at all, and so we ended up implementing the worker side of this in TC BPF.
If only fearsome-bagel is supposed to access serf, can serf do a reverse lookup on the source IP and then make sure it matches regexp /^fearsome-bagel-\d+\.internal$/?
> I love the writing style here. It seems to have the passion and perspective of a founder but is super technical. I’m curious how writing like this gets executed at a company like this. What’s the workflow?
I do believe that the dual goals of network identity and scalability are difficult to solve for simultaneously (in the old world, we relied on subnets, but this is the opposite of multi-tenant container hosts..) so i think we want to have simple physical networks with routing and then some other means of trusted network identity, which leads you to crypto-derived solutions (really, it seems like wg and IPSec are the biggest contenders here and both have pros and cons)
Anyway, seeing your post was validating about this line of reasoning and I am curious to follow along to see how you are thinking about it more.
>North America Europe 100GB per month free $0.02 per GB
>India 30 GB per month free $0.12 per GB
How come India is so much more expensive?
Is this how Jio manages to keep consumer costs down, by making service providers pay the bill?
[1] https://fly.io/docs/about/pricing/#outbound-data-transfer
and vicious cycle (lack of demand -> cant sustain to pay for equipment and maintenance cost -> rise higher price to compensate -> be more expensive -> subscribers cant pay -> quit -> lesser demand -> rinse and repeat)
The tldr: we don't have heavy usage in India. When we do, it'll be about half that price.
The right thing to do would be to write two blog posts, but there's no better way to get me not to write something than to tell me I have to write two somethings.