Hacker News Reader

Comment by iajrz | Hacker News Reader

When you say "in place" you mean the data is still on the NIC? Or is the data 'elsewhere in RAM'?

I'm curious because this would be the difference between "scale with more cores" or "scale with more NICs".

Generally, how it works is that there will be multiple hardware-managed ring buffers allocated in host memory, which will point to packet data which also is allocated in host memory. When a packet is received, the NIC will DMA the packet into host memory, update the descriptors in the ring buffers, and trigger an interrupt.[1] That is the point at which the kernel driver can access the packet (in host memory). See: https://www.intel.com/content/dam/www/public/us/en/documents... (section 8.3.3). By "in place" I mean it can be accessed from the location where the NIC put it.

In terms of scaling with multiple CPUs, generally these days the NIC will be able to manage multiple separate ring buffers. Ring buffers can be dedicated to specific CPUs, and the NIC can distribute packets among the ring buffers (in hardware) by hashing on the packet's IP 5-tuple.[2] So long as you have packets that are distributed evenly by that mechanism (not the case when, e.g. the packets all belong to the same connection), you can scale with multiple CPUs. (Presumably the NIC can perform the distribution process in hardware at line rate.)

[1] At high packet rates, there might be an interrupt less frequently than for every packet.

[2] See section 7.1.8 of the linked manual.

If I read that correctly the data is still on the NIC. The CPU receives an interrupt - notifying it of a new packet, which it then decides to drop? Thus the CPU is the one dropping packets, but the data (most of it at least) is still on the NIC.

Please correct me if I'm wrong.

This all happens after the packet is transferred over PCIe to the CPU/memory (even the interrupt happens after DMA transfer). Otherwise the XDP code wouldn't have access to the packet to filter it; the example doesn't just drop all packets, it drops all packets matching a /24 subnet which happens to be all packets here ;)

Ah ok! I was kinda confused by that xD

Reply on news.ycombinator.com