How to drop 10M packets per second | Hacker News Reader

How to drop 10M packets per second

blog.cloudflare.com

194 pointssigna1129 comments

I recently heard about VPP for the first time; VPP is one of those user-space networking frameworks. Apparently it's the fastest thing currently around by a large margin ("10 mpps per core? Too slow for me!"); it's alluded that Cisco's ASR9000 routers use the same core tech to do all routing in software.

Important note, DPDK and VPP are not faster by the virtue of being based in userspace. It is best to say that they just provide fastest way in between interrupt handler, and the piece of userspace code. The application will then have to reimplement most of the network stack functions that run in kernel in itself. The speedup is there because the application author can freely slim down the network stack to minimal. Secondly to that DPDK took some deliberation on better driver buffer management, ensuring zero copy path, interrupt and syscall efficiency.

AFAIK, you don't even run interrupts off the NIC if you're doing DPDK; the overhead is too high. You poll instead, and just dedicate a core.

I'm unsure. What I'm imagining is that NICs for which DPDK is made have option to do interrupts like "hey, we have just run out of 50% of buffer! Driver, go offload packets to RAM in bulk now"

DPDK's drivers are (as far as I know?) all poll-mode: https://doc.dpdk.org/guides/prog_guide/poll_mode_drv.html

Correct

> It is best to say that they just provide fastest way in between interrupt handler

I thought DPDK and friends don't use interrupts and instead map the buffer areas (for ring buffers etc.) used by the NIC into their process address space, directly polling those structures.

I have to check myself. I can't quite remember, but them simply polling dma buffers is also a possibility.

What confuses me is that deep in my memory something tells that DPDK had some kind of "deterministic mode" with guaranteed "zero drop at below n packets per n nanoseconds" that is driven by ring buffer "rollover sense" interrupts.

Or may be it just uses those to set the poll rate.

Right, but vpp tries to solve, at least part of this, by implementing many of the user space protocols. The only concern I have is I don't know how Cisco plans to monetize it, or why they're paying people to work on it. It seems very anti-cisco.

VPP is ~15 years old and Cisco already monetized it in their own products.

VPP is pretty neat. I just set it up as a router/bridge on my home network. (Upgraded to 10G and Linux can’t route or bridge 10G in software with small packets.) OpenSUSE packages everything and the instructions are great. Very easy to set up.

There is really no other option for line rate 10G packet handling in software. Linux can handle maybe 1.5 Mpps per core. That’s barely line rate for 1G. Multiple cores speed that up, but for sequencing purposes packets are usually distributed to different cores by hashing the IP 5-tuple, which means that more cores don’t help for a single connection (or a DDOS like the one in the article).

Neat. Based on a brief Google search, it appears to use DPDK if available.

I’m going to switch my home router to use it, it seems to support PPPoE: https://docs.fd.io/vpp/18.07/clicmd_src_plugins_pppoe.html

And works with FRR: https://github.com/FRRouting/frr/wiki/Alternate-forwarding-p...

More docs on VPP can be found here, including installation, getting started, configuration and FAQ: https://wiki.fd.io/view/VPP

There’s even a Kubernetes networking plugin for VPP: https://github.com/contiv/vpp

>Linux can handle maybe 1.5 Mpps per core. That’s barely line rate for 1G.

At what packet size? At 1500B, 10Gbps is only 800Kpps.

In my experience working a large network, 300-600B seems to be the average. Worst case, that's ~420Kpps.

As far as I've heard the term used in a packet-switched networking context, "line rate" implies the smallest valid packet size.

> There is really no other option for line rate 10G packet handling in software.

DPDK is an example of another option.

VPP uses DPDK

> VPP is one of those user-space networking frameworks.

oh yes, vpp is builds on what is provided by dpdk.

"With XDP we can drop 10 million packets per second on a single CPU." is not really correct, the packets are being dropped on the network card which has its own processor. The packets never get transferred to the main CPU.

That’s not true. While there are NICS that can drop packets in hardware (e.g. Chelsio) that is not what XDP does. XDP runs eBPF programs in a driver-implemented recieve callback. The recieve callback happens at the lowest level of the Linux network stack, when an interrupt is delivered to a processor and a packet descriptor is queued by the NIC, but before the generic packet handling metadata (SKB) is allocated. So XDP operates on the packet in place, but the code runs on the host CPU.

XDP can be offloaded to the NIC [1]. This is what I took the quote "the userspace loader, which loads the code onto the right network card and manages it" to mean.

But I think you might be right in this case, it seems more likely that the XDP processing is being done by the network card driver.

[1] https://cilium.readthedocs.io/en/latest/bpf/#iproute2

XDP can run in the driver in the "normal" mode, so it's on host CPU. But for some NICs like Netronome nfp, the kernel can JIT the BPF program into the instruction set for nfp and then run 'offloaded' on the card directly, so not even affecting host CPU in any way. See also here for a recent blog post: https://www.netronome.com/blog/bpf-ebpf-xdp-and-bpfilter-wha... So both is possible, but even with XDP running on host CPU like in Cloudflare case above you get excellent performance numbers.

When you say "in place" you mean the data is still on the NIC? Or is the data 'elsewhere in RAM'?

I'm curious because this would be the difference between "scale with more cores" or "scale with more NICs".

Generally, how it works is that there will be multiple hardware-managed ring buffers allocated in host memory, which will point to packet data which also is allocated in host memory. When a packet is received, the NIC will DMA the packet into host memory, update the descriptors in the ring buffers, and trigger an interrupt.[1] That is the point at which the kernel driver can access the packet (in host memory). See: https://www.intel.com/content/dam/www/public/us/en/documents... (section 8.3.3). By "in place" I mean it can be accessed from the location where the NIC put it.

In terms of scaling with multiple CPUs, generally these days the NIC will be able to manage multiple separate ring buffers. Ring buffers can be dedicated to specific CPUs, and the NIC can distribute packets among the ring buffers (in hardware) by hashing on the packet's IP 5-tuple.[2] So long as you have packets that are distributed evenly by that mechanism (not the case when, e.g. the packets all belong to the same connection), you can scale with multiple CPUs. (Presumably the NIC can perform the distribution process in hardware at line rate.)

[1] At high packet rates, there might be an interrupt less frequently than for every packet.

[2] See section 7.1.8 of the linked manual.

If I read that correctly the data is still on the NIC. The CPU receives an interrupt - notifying it of a new packet, which it then decides to drop? Thus the CPU is the one dropping packets, but the data (most of it at least) is still on the NIC.

Please correct me if I'm wrong.

This all happens after the packet is transferred over PCIe to the CPU/memory (even the interrupt happens after DMA transfer). Otherwise the XDP code wouldn't have access to the packet to filter it; the example doesn't just drop all packets, it drops all packets matching a /24 subnet which happens to be all packets here ;)

Ah ok! I was kinda confused by that xD

Exactly. And thus this approach does not scale with more cores at all.

Let me show how to render 100B polygons per second on a single cpu by using a gpu to do all the work... Argh.

now find a way for me to enter an unknown/untrusted website that is using your "ddos protection" without having to allow inline javascript to execute

because as you clearly know/demonstrate, the web can't be trusted by default

I'm an electrician. I can help you drop packets really quick. Available for hire.

Reply on news.ycombinator.com