Hacker News Reader

Comment by rayiner | Hacker News Reader

That’s not true. While there are NICS that can drop packets in hardware (e.g. Chelsio) that is not what XDP does. XDP runs eBPF programs in a driver-implemented recieve callback. The recieve callback happens at the lowest level of the Linux network stack, when an interrupt is delivered to a processor and a packet descriptor is queued by the NIC, but before the generic packet handling metadata (SKB) is allocated. So XDP operates on the packet in place, but the code runs on the host CPU.

View on HN

XDP can be offloaded to the NIC [1]. This is what I took the quote "the userspace loader, which loads the code onto the right network card and manages it" to mean.

But I think you might be right in this case, it seems more likely that the XDP processing is being done by the network card driver.

[1] https://cilium.readthedocs.io/en/latest/bpf/#iproute2

XDP can run in the driver in the "normal" mode, so it's on host CPU. But for some NICs like Netronome nfp, the kernel can JIT the BPF program into the instruction set for nfp and then run 'offloaded' on the card directly, so not even affecting host CPU in any way. See also here for a recent blog post: https://www.netronome.com/blog/bpf-ebpf-xdp-and-bpfilter-wha... So both is possible, but even with XDP running on host CPU like in Cloudflare case above you get excellent performance numbers.

When you say "in place" you mean the data is still on the NIC? Or is the data 'elsewhere in RAM'?

I'm curious because this would be the difference between "scale with more cores" or "scale with more NICs".

Generally, how it works is that there will be multiple hardware-managed ring buffers allocated in host memory, which will point to packet data which also is allocated in host memory. When a packet is received, the NIC will DMA the packet into host memory, update the descriptors in the ring buffers, and trigger an interrupt.[1] That is the point at which the kernel driver can access the packet (in host memory). See: https://www.intel.com/content/dam/www/public/us/en/documents... (section 8.3.3). By "in place" I mean it can be accessed from the location where the NIC put it.

In terms of scaling with multiple CPUs, generally these days the NIC will be able to manage multiple separate ring buffers. Ring buffers can be dedicated to specific CPUs, and the NIC can distribute packets among the ring buffers (in hardware) by hashing on the packet's IP 5-tuple.[2] So long as you have packets that are distributed evenly by that mechanism (not the case when, e.g. the packets all belong to the same connection), you can scale with multiple CPUs. (Presumably the NIC can perform the distribution process in hardware at line rate.)

[1] At high packet rates, there might be an interrupt less frequently than for every packet.

[2] See section 7.1.8 of the linked manual.

If I read that correctly the data is still on the NIC. The CPU receives an interrupt - notifying it of a new packet, which it then decides to drop? Thus the CPU is the one dropping packets, but the data (most of it at least) is still on the NIC.

Please correct me if I'm wrong.

This all happens after the packet is transferred over PCIe to the CPU/memory (even the interrupt happens after DMA transfer). Otherwise the XDP code wouldn't have access to the packet to filter it; the example doesn't just drop all packets, it drops all packets matching a /24 subnet which happens to be all packets here ;)

Ah ok! I was kinda confused by that xD

Reply on news.ycombinator.com