Hacker News Reader

Comment by inigyou | Hacker News Reader

I don't understand how a product as popular as Tailscale can get this far while dropping certain ordinary types of packets.

It is impossible to parse the UDP or TCP port number out of a fragment. This is surely the reason the ACL module entirely rejects them. TCP will adjust it's segment size based on PMTUD so as to not require fragmentation. This is why it hasn't been noticed so far. But fragmented UDP packets are a corner case of normal behavior and it boggles the mind that someone could just decide to completely drop them.

UDP fragment filtering could be implemented by a global fragments on/off setting (works for "allow everything" = fragments on, cautious = fragments off) or by blocking the first fragment which includes the port number (and blocking it if the port number is split across fragments which I think is technically allowed but completely abnormal).

View on HN

> I don't understand how a product as popular as Tailscale can get this far while dropping certain ordinary types of packets.

I’d venture to guess based on this outcome that fragmented UDP over IPv6 isn’t really an ordinary occurrence. Given the preponderance of HTTPS traffic, the aversion to fragmentation in IPv6, and the weird corner case of there being a hardcoded packet size in webrtc, it’s reasonable to assume that this is a corner case.

A good one to be aware of, but not common.

Would agree it's uncommon in general traffic. Rare conditions [webrtc-rs, 1280 class tunnel / tailscale, and ipv6 pair] but deadly when they are met since every connection silently fails. That's what made it worth chasing down for 2 weeks [and good for sleuthing :)].

It's a corner case of ordinary traffic, since all TCP apps and most UDP apps adapt to PMTU, but fragmentation is there for those that don't. It's not like something you can only get by generating malicious traffic intentionally.

Welcome to networking mistakes, I guess. I can't remember the specifics but I once encountered a router that would drop traffic that looked like encapsulated TCP at a certain offset, or something like that. They couldn't fix it because the behavior was hardwired. I knew of it because I worked with the firmware team.

Factorio discovered that UDP packets with a checksum of 0x0000 get dropped by some devices.

Dropping fragments is a pretty normal thing to do in a lot of places. If you have a stateful firewall, you can't tell if a fragment is viable until you reassemble it, and reassembly is unreasonably expensive, so dropping fragments it is.

Personally, I prefer to go ahead and reassemble, but with a very minimal reassembly buffer.

Very few packets get fragmented, so if you have more than 16 fragments in your reassembly buffer, you're probably being ddosed and you can toss them. OTOH, if you have a 16 deep reassembly buffer, you're probably more generous than most services that have no buffer for reassembly.

It's not what the RFCs say to do, but the IPv6 RFCs are like 30 years old, and the IPv4 RFCs even older. They were written in a different time for an internet that was less adversarial; some things don't make sense to keep doing.

Author here,

Agreed. The port-number point is the most plausible rationale I've heard, more convincing than the RFC line in their source comment. The historical fix for "can't classify fragments" was virtual reassembly or flow tracking [conntrack on linux, scrub in pf], so dropping them outright punts past known prior approaches. Even your lighter idea would have saved us: a first-fragment match would have let our pair through.

We've reported upstream to both projects, tailscale/tailscale#20083 and webrtc-rs/webrtc#806, and webrtc-rs already invited a PR.

You are shadowbanned.

Reply on news.ycombinator.com