Comment by kevincox | Hacker News Reader

Comment by kevincox | Hacker News Reader

> the endpoints need to use large buffers to hold a copy of all the unacknowledged data, as is required by the TCP protocol.

It makes me wonder if anyone has tried to break down the layers to optimize this. In the fairly common case of serving a file off of long-term storage you can jut fetch the old data if needed (likely from the page cache anyways, but still better than duplicating it) and some encryption algorithms are seekable so you can redo the encryption as well.

Right now the kernel doesn't really have a choice but to buffer all unacknowledged data as the UNIX socket API has no provision for requesting old data from the writer. But a smarter API could put the application in charge of making that data available as required and avoid the need for extra copies in common cases.

I know that Netflix did lots of work with the FreeBSD kernel for file to socket and eventually adding in-kernel TLS to remove user space from the equation. But I don't know if they went as far to remove the socket buffers.

This is an "old" problem that has historically been addressed through things like "Performance Enhancing Proxies (PEPs)" that are defined in RFC 3135 and RFC 3449. (https://en.wikipedia.org/wiki/Performance-enhancing_proxy)

In internet-style communications, such as routing IP traffic over satellite links to low-earth-orbit or GEO, with much longer round-trip times, link latency is substantially higher than most terrestrial wired or wireless applications and acknowledgements required as part of TCP take much longer to facilitate. PEPs as an example augment the connection allowing end-user/client devices in the network with inline-PEPs to retain their normal network settings and perform the task of running or starting sessions with higher TCP-window sizes, as a method for improving overall throughput.

The utility of a PEP, or PEP-acting device goes up when you imagine multiple devices, or a network of devices attached to a satellite communications terminal for WAN/backhaul connections as the link's performance can be managed at one point versus on all downstream client devices.

TIL that that's what they're called. Thank you!

Do you know if they became obsolete due to modern TCP stacks handling LFNs better or for some other reason?

I could imagine them being quite useful for high-loss, high-latency paths (i.e. in addition to LFNs in conjunction with poorly tuned TCP implementation), but most wireless protocols I know (802.11, GPRS and beyond etc.) just implement an ARQ process that masks the loss at the packet or lower layer.

So maybe between that and LFN-aware TCPs, there wasn't much left for them to improve to justify the additional complexity?

It's been a long time since I've paid attention to that world, but AFAIK, PEPs are still used and essential equipment for internet-style communication (i.e. TCP over IP) via GEO satellites.

It looks like in this 2022 blog post evaluating latency and throughput over Starlink, they concluded that PEPs were not being used in the Starlink network (and probably unnecessary) due to the lower latency characteristics from use of LEO satellites. They also mention that PEPs are (still) commonly employed by GEO-satcom based operators.

https://blog.apnic.net/2022/11/28/fact-checking-starlinks-pe...

> they concluded that PEPs were not being used in the Starlink network

That makes sense, given that they're probably most useful for high-latency networks. But what I find quite surprising is that Starlink does nothing about the 1-2% packet loss, as described in TFA; I'd really have expected them to fix that using an ARQ at a lower layer.

Then again, maybe that's a blessing – indiscriminate ARQs like that would be terrible for time critical things like A/V, which can usually tolerate packet loss much better than ARQ-induced jitter.

Thinking about it, that actually strengthens the case for PEPs: They could improve TCP performance (and maybe things like QUIC?), while leaving non-stream oriented things (like non-QUIC UDP) alone.

Maybe Starlink just expects BBR to eventually become the dominant TCP congestion control algorithm and the problem to solve itself that way?

> It makes me wonder if anyone has tried to break down the layers to optimize this.

Yep. There was a bunch of proxy servers that optimized HTTP for satellite service. I used Globax back in the day to speed up one-way satellite service: https://web.archive.org/web/20040602203838/http://globax.inf...

Back then traffic was around 10 cents per megabyte in my city, so satellite service was a good way to shave off these costs.

There aren't necessarily extra copies even when just using TCP, thanks to sendfile(2) and similar mechanisms.

Buffer size isn't that much of an issue either, given the relatively low latencies involved and that you can indicate which parts exactly are missing pretty accurately these days with selective TCP acknowledgements, so you'll need at most a few round trips to identify these to the sender and eventually receive them.

Practically, you'll probably not see much loss anyway, for better or worse: TCP historically interpreted packet loss as congestion, instead of actual non-congestion-induced loss on the physical layer. This is why most lower-layer protocols with higher error rates than Ethernet usually implement some sort of lower-layer ARQ to present themselves as "Ethernet-like" to upper layers in terms of error/loss rate, and this in turn has made loss-tolerant TCP (such as BBR, as described in the article) less of a research priority, I believe.

Reply on news.ycombinator.com