How to receive a million packets per second (2015)
blog.cloudflare.com
blog.cloudflare.com
At 10Gbps, and hundred-odd byte packets as are common on finance multicast feeds, I get 73ns per packet. The amount that can be done in that time is limited. Typically, you just dump them in a big-ass ring buffer and let processes on a bunch of other cores pick out interesting ones to spend more time on.
The way to get the packets is usually with some proprietary kernel-bypass library (some have success with DPDK), and a very carefully isolated core that the kernel is kept from interrupting, and just spins moving packets. NIC hardware-level ring buffers are usually pretty small, so pauses make for drops.
Many NICs can be persuaded to filter packets out to multiple ring buffers, and then you can get multiple cores picking them off; that is necessary when you get to 40 and 100 Gbps. Multiple rings even with one core multiplies headroom, making occasional housekeeping without drops possible.
Background is that I did some work connecting to a few smaller exchanges, but we never ever thought we’d ever need to go past “just” 20mbps.
Nobody said finance invented anything. But high performance demands produce similar architectures everywhere.
Well, yes. In March, the per-day volumes went to more than double the previous peak.
That said, traffic is bursty, particularly at market open and close, and latency matters, so you have to statically provision for peak traffic.
Have you tried out the new eBPF libraries and kTLS?
eBPF and kTLS tend to involve the kernel in your business. But, eBPF can put code directly on certain NICs (e.g. Netronome), writing directly to user process memory. Then there is no need to pin or isolate, because the action is on the NIC, out of reach.
One question I don't see addressed in that thread.How do you deal with failures of the node? Do you use active-active type replication or some other strategy?
It is usually best to keep systems as decoupled as possible, and to make possible responses to failure as few and simple as possible.
5 years later I have servers that can do 2 million (multi process) without messing with Numa or nic settings.
A few years ago I worked on the upload service within the YouTube Android app. Shortly after YouTube started using QUIC for video streaming I decided it would be interesting to experiment using QUIC for mobile uploads; my thesis being that its BBR implementation might lead to better throughput on lossy wireless connections and therefore faster mobile uploads. I discovered though when tethered to my dev workstation I wasn't able to break ~100mbit while the native TCP stack was able to saturate the USB-C gigabit ethernet adapter I was using. This problem got the attention of the QUIC folks, which lead to a more detailed investigation than I was capable of and eventually to kernel patches described here: http://vger.kernel.org/lpc_net2018_talks/willemdebruijn-lpc2...
I'd like to think my little project was the major motivator for that work but I'm pretty sure the server efficiency gains described in that paper were a much stronger motivation. :)
Specifically, if QUIC is to have high performance and high power efficiency, just batching the packets isn't enough. Think of a usecase - a CDN wants to be able to send a block of data from disk to a client. It should be able to direct that data from the SSD to the network card without the CPU touching the data at all.
On the receive side, a mobile device should be able to download a big file to ram or flash with the main CPU asleep most of the time.
That means the network device should be encrypting the packets, doing the pacing, doing the retries, etc.
It may be that a quic specific set of options or syscalls is best for that, or it might be that the application should provide a "this is how to send my data" set of bytecode which the network hardware executes repeatedly.
But just batching UDP packets is a half way measure which isn't in the direction of the final goal.
Right, that paper doesn't introduce life-shattering innovation but rather just fixes a long-standing issue with UDP performance in the Linux kernel. That ain't nothing.
A large motivator to develop and deploy QUIC is that by implementing much of the stack as a userspace library the protocol can be iterated on at a pace impossible with TCP. BBR congestion control experiments are a great example of that; there's enormous value in quick iteration, even at the cost of compute efficiency.
I agree with your sentiment though; I have to imagine eventually the protocol will settle down and hardware acceleration will become a thing - but imho not to the degree you propose.
$ ./local_thr tcp://192.168.1.123:5678 100 1000000
message size: 100 [B]
message count: 1000000
mean throughput: 1143708 [msg/s]
mean throughput: 914.966 [Mb/s]