Hacker News Reader

Comment by Animats | Hacker News Reader

What is UDP offload going to do? UDP barely does anything but queue and copy.

Linux scheduling from packet-received to thread has control is not real-time, and if the CPUs are busy, may be rather slow. That's probably part of the bottleneck.

The embarrassing thing is that QUIC, even in Google's own benchmarks, only improved performance by about 10%. The added complexity probably isn't worth the trouble. However, it gave Google control of more of the stack, which may have been the real motivation.

View on HN

Last I looked (several months ago), Linux's UDP stack did not seemed well tuned in its memory management accounting.

For background, the mental model of what receiving network data looks like in userspace is almost completely backwards compared to how general-purpose kernel network receive actually works. User code thinks it allocates a buffer (per-socket or perhaps a fancier io_uring scheme), then receives packets into that buffer, then processes them.

The kernel is the other way around. The kernel allocates buffers and feeds pointers to those buffers to the NIC. The NIC receives packets and DMAs them into the buffers, then tells the kernel. But the NIC and the kernel have absolutely no concept of which socket those buffers belong to until after they are DMAed into the buffers. So the kernel cannot possibly map received packets to the actual recipient's memory. So instead, after identifying who owns a received packet, the kernel retroactively charges the recipient for the memory. This happens on a per-packet basis, it involves per-socket and cgroup accounting, and there is no support for having a socket "pre-allocate" this memory in advance of receiving a packet. So the accounting is gnarly, involves atomic operations, and seems quite unlikely to win any speed awards. On a very cursory inspection, the TCP code seemed better tuned, and it possibly also won by generally handling more bytes per operation.

Keep in mind that the kernel can't copy data to application memory synchronously -- the application memory might be paged out when a packet shows up. So instead the whole charging dance above happens immediately when a packet is received, and the data is copied later on.

For quite a long time, I've thought it would be nifty if there was a NIC that kept received data in its own RAM and then allowed it to be efficiently DMAed to application memory when the application was ready for it. In essence, a lot of the accounting and memory management logic could move out of the kernel into the NIC. I'm not aware of anyone doing this.

> For quite a long time, I've thought it would be nifty if there was a NIC that kept received data in its own RAM and then allowed it to be efficiently DMAed to application memory when the application was ready for it.

I wonder if we could do a more advanced version of receive-packet steering that sufficiently identifies packets as definitely for a given process and DMAs them directly to that process's pre-provided buffers for later notification? In particular, can we offload enough information to a smart NIC that it can identify where something should be DMAed to?

Most advanced NICs support flow steering, which makes the NIC write to different buffers depending on the target port.

In practice though, you only have a limited amount of these buffers, and it causes complications if multiple processes need to consume the same multicast.

Multicast may well be shitcanned to an expensive slow path, given that multicast is rarely used for high bandwidth scenarios, especially when multiple processes need to receive the same packet.

The main real use of multicast I’ve seen is pretty high packet rate. High frequency traders get multicast feeds of tick data from the exchange.

multicast is precisely used for low-latency high-throughput message buses.

With multiple processes listening for the data? I think that's a market niche. In terms of billions of devices, multicast is mostly used for zero-config service discovery. I am not saying there isn't a market for high-bandwidth multicast, I am stating that for the vast majority of software deployments, multi-cast performance is not an issue. For whatever deployments it is an issue, they can specialize. And, as in the sibling comment mentions, people who need breakneck speeds have already proven that they can create a market for themselves.

That's not a market niche, that's the normal mode of operation of a message bus.

The cloud doesn't implement multicast, but that doesn't mean it doesn't get used by people that build non-Internet networks and applications.

I will reiterate that most if not all desktop/laptop class x86 hardware as it is deployed in the wild does not care about multicast performance.

Those are Internet devices.

I don’t think the result would be compatible with the socket or io_uring API, but maybe io_uring could be extended a bit. Basically the kernel would opportunistically program a “flow director” or similar rule to send packets to special rx queue, and that queue would point to (pinned) application memory. Getting this to be compatible with iptables/nftables would be a mess or maybe entirely impossible.

I’ve never seen the accelerated steering stuff work well in practice, sadly. The code is messy, the diagnostics are basically nonexistent, and it’s not clear to me that many drivers support it well.

RDMA is common for high performance applications but it doesn't work over the Internet.

> RDMA is common for high performance applications but it doesn't work over the Internet.

RoCEv2 is routable.

* https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet

* https://docs.nvidia.com/networking/display/winofv55053000/ro...

Of course you're going to get horrible latency because of speed-of-light limitations, so the definition of "work" may be weak, but data should be able to be transmitted.

It's a good thing the NIC is connected over pcie then.

You can do GPUdirect over the Internet without RDMA though.

GPUDirect relies on the PeerDirect extensions for RDMA and are thus an extension to the RDMA verbs, not a separate an independent thing that works without RDMA.

Again, you can do what I said. You may be using different terminology, but you can do GPUdirect in dpdk without rdma

You can read/write to GPU buffers with gpudev in DPDK yes. It also uses some of the infrastructure that powers GPUDirect (namely the page pinning and address translation). Because you can use the addressable memory in DPDK buffer steering you can have the NIC DMA to/from the GPU and then have a GPU kernel coordinate with your DPDK application. This will be pretty fast on a good lossless datacentre network but probably pretty awful over the Internet. In the DC though it will be beaten by real GPUDirect on RDMA naturally as you don't need the DPDK coordinator and all tx/rx can be driven by the GPU kernel instead.

This isn't GPUDirect though, that is an actual product.

This is GPUDirect. GPUDirect is the technology that enables any third-party device to talk to a GPU (like a NIC).

> but probably pretty awful over the Internet. In the DC though it will be beaten by real GPUDirect on RDMA naturally

It's being used in many places successfully over the internet. RDMA is fine, but completely breaks the abstraction of services. In many places you do not want to know who is sending or what address to send/receive.

Have you looked into NVIDIA VMA?

https://docs.nvidia.com/networking/display/vmav9860/introduc...

Why don't we eliminate the initial step of an app reserving a buffer, keep each packet in its own buffer, and once the socket it belongs to is identified hand a pointer and ownership of that buffer back to the app? If buffers can be of fixed (max) size, you could still allow the NIC to fill a bunch of them in one go.

Presuming that this is a server that has One (public) Job, couldn't you:

1. dedicate a NIC to the application;

2. and have the userland app open a packet socket against the NIC, to drink from its firehose through MMIO against the kernel's own NIC DMA buffer;

...all without involving the kernel TCP/IP (or in this case, UDP/IP) stack, and any of the accounting logic squirreled away in there?

(You can also throw in a BPF filter here, to drop everything except UDP packets with the expected specified ip:port — but if you're already doing more packet validation at the app level, you may as well just take the whole firehose of packets and validate them for being targeted at the app at the same time that they're validated for their L7 structure.)

I think DPDK does something like this. The NIC is programmed to aim the packets in question at a specific hardware receive queue, and that queue is entirely owned by a userspace program.

A lot of high end NICs support moderately complex receive queue selection rules.

> 1. dedicate a NIC to the application;

you need to respond to ICMPs which have different proto/header number than UDP or TCP.

I mean, under the scheme I outlined, the kernel is still going to do that by default. It's not that the NIC's driver is overridden or anything; the kernel would still be reading the receive buffer from this NIC and triggering per-packet handling — and thus triggering default kernel response-handling where applicable (and so responding to e.g. ICMP ARP messages correctly.)

The only thing that's different here, is that there are no active TCP or UDP listening sockets bound to the NIC — so when the kernel is scanning the receive buffer to decide what to do with packets, and it sees a TCP or UDP packet, it's going to look at its connection-state table for that protocol+interface, realize it's empty, and drop the packet for lack of consumer, rather than doing any further logic to it. (It'll bump the "dropped packets" counter, I suppose, but that's it.)

But, since there is a packet socket open against the NIC, then before it does anything with the packet, it's going to copy every packet it receives into that packet socket's (userspace-shared) receive-buffer mmap region.

UDP offload gets you implicitly today:

- 64 packets per syscall, which is enough data to amortize the syscall overhead - a single packet is not.

- UDP offload optionally lets you defer checksum computation, often offloading it to hardware.

- UDP offload lets you skip/reuse route lookups for subsequent packets in a bundle.

What UDP offload is no good for though, is large scale servers - the current APIs only work when the incoming packet chains neatly organize into batches per peer socket. If you have many thousands of active sockets you’ll stop having full bundles and the overhead starts sneaking back in. As I said in another thread, we really need a replacement for the BSD APIs here, they just don’t scale for modern hardware constraints and software needs - much too expensive per packet.

In my head the main benefit of QUIC was always multipath, aka the ability to switch interfaces on demand without losing the connection. There's MPTCP but who knows how viable it is.

I always thought the main benefit of QUIC was to encrypt the important part of the transport header, so endpoints control their own destiny, not some middle device.

If I had a dollar for every firewall vendor who thought dropping TCP retransmissions or TCP Reset was a good idea...

Apple's Siri is using MPTCP, so it is presumably viable.

It requires explicit backend support, and Apple supports it for many of their services, but I've never seen another public API that does. Anyone have any examples?

Last I looked into this (many years), ELB/GLBs didn't support it on AWS/GCP respectively. That prevented us from further considering implementing it at the time (mobile app -> AWS-hosted EC2 instances behind an ELB).

Not sure if that's changed, but at the time it wasn't worth having to consider rolling our own LBs.

To answer your original question, no, I haven't (knowingly) seen it on any public APIs.

Is that actually implemented and working in practice? My connection still hangs whenever my wifi goes out of range...

Mptcp sees use in the Telco space, so they probably know.

Ditching head of line blocking is potentially a big win, but I really wish it wouldn't have come with so much complexity.

Among other things, GRO (receive offloading) means you can get more data off of the network card in fewer operations.

Linux has receive packet steering, which can help with getting packets from the network card to the right CPU and the right userspace thread without moving from one CPU's cache to another.

RPS is just software RSS.

You mean Receive Flow Steering, and RFS can only control RPS, so to do it in hardware you actually mean Accelerated RFS (which requires a pretty fancy NIC these days).

Even ignoring the hardware requirement, unfortunately it's not that simple. I find results vary wildly whether you should put process and softirq on the same CPU core (sharing L1 and L2) or just on the same CPU socket (sharing L3 but don't constantly blow out L1/L2).

Eric Dumazet said years ago at a Netdev.conf that L1 cache sizes have really not kept up with reality. That matches my experience.

QUIC doing so much in userspace adds another class of application which has a so-far uncommon design pattern.

I don't think it's possible to say whether any QUIC application benefits from RFS or not.

> What is UDP offload going to do?

Handling ACK packets in kernelspace would be one thing - helping for example RTT estimation. With userspace stack ACK's are handled in application and are subject to scheduler, suffering a lot on a loaded system.

There are no ACKs inherent in the UDP protocol, so "UDP offload" is not where the savings are. There are ACKs in the QUIC protocol and they are carried by UDP datagrams which need to make their way up to user land to be processed, and this is the crux of the issue.

What is needed is for QUIC offload to be invented/supported by HW so that most of the high-frequency/tiny-packet processing happens there, just as it does today for TCP offload. TCP large-send and large-receive offload is what is responsible for all the CPU savings as the application deals in 64KB or larger send/receives and the segmentation and receive coalescing all happen in hardware before an interrupt is even generated to involve the kernel, let alone userland.

Bulk throughout isn't on par with TLS mainly because NICs with dedicated hardware for QUIC offload aren't commercially available (yet). Latency is undoubtedly better - the 1-RTT QUIC handshake substantially reduces time-to-first-byte compared to TLS.

I think one of the original drivers was the ability to quickly tweak parameters, after Linux rejected what I think was userspace adjustment of window sizing to be more aggressive than the default.

The Linux maintainers didn't want to be responsible for congestion collapse, but UDP lets you spray packets from userspace, so Google went with that.

Reply on news.ycombinator.com