Hacker News Reader

Comment by contrarian_ | Hacker News Reader

Sounds like servers handling lots of small UDP packets would be hit pretty hard.

Applications like this where the syscall overhead (and latency) starts to be a significant factor in processing time and latency have moved to userland drivers anyway:

DPDK for 10-100 Gbps networking: https://dpdk.org/

SPDK for NVMe storage: http://www.spdk.io/

The queuing and balancing stuff the kernel does makes sense for spinning rust harddisks and residential networking, but when the underlying hardware is so fast that nothing is ever queued, really what are you doing. At 100 Gbps line speed, a 1518 byte packet takes all of ~ 120ns to transmit, or about 360 clock cycles for a 3 GHz processor.

Taking control over network cards in user space seems doable nowdays. There was a talk about doing such drivers with IOMMU/DMA at CCC: https://media.ccc.de/v/34c3-9159-demystifying_network_cards

User-space drivers have been doable for a while, and dpdk[1] is definitely worth a check. There's also some manufacturers[2] that only does user-space drivers for their high-performance cards (e.g. 4x10Gb/s, 2x40Gb/s, 2x100Gb/s cards). Being designed with this in mind helps performance a lot.

1: https://dpdk.org/ 2: http://www.napatech.com/

> Applications like this where the syscall overhead (and latency) starts to be a significant factor in processing time and latency have moved to userland drivers anyway:

I would personally think that is worse, though please correct me if I'm wrong. The userland driver will run with an isolated PT like any other userland process won't it? If so, it will suffer the same slowdown that every other process now has every-time it has to communicate with the kernel, which I would think would be a lot for a driver.

It's counter intuitive at first, but the key to understand how this works is that while you can use an MMU to assign chunks of physical memory to a process, you can of course also just use the MMU to assign the memory mapped IO registers of say a PCI express peripheral to a process.

That is in a nutshell what a "userland driver" is. It's not too far removed from poking the parallel port at 0x378 on your DOS computer :)

A user-space driver doesn't communicate with the kernel. It is assigned DMA buffers, and communicates with the NIC solely through reading and writing to shared memory buffers.

Even before this fix, the benefits were massive, as sending a buffer was just writing to some memory, rather than syscalls and copies galore.

Reduce number of syscalls by using sendmmsg()/recvmmsg() to batch together multiple packets per syscall?

Anyone run any DNS server benchmarks, esp BIND and PowerDNS?

Reply on news.ycombinator.com