How to receive a million packets per second

[1] https://groups.google.com/forum/m/#!topic/snabb-devel/_vKQgC...

Apparently you can also use the SO_RCVTIMEO socket option, which is a way to specify a timeout for all receive operations on the socket.

Netmap is commonly used in HFT as well as packet filtering applications. I believe Verisign is running some of the root DNS servers with netmap as well, getting millions of connections per second.

> two cores busy with handling RX queues, and third running the application, it's possible to get ~650k pps

That's ~200k pps per core, so 4x the initial bet.

I don't think there is any particular problem with Linux. The core problem of "slow" performance is mostly due to limitations of BSD API (recvmmsg is a good example of a workaround), and due to the feature richness of Linux.

Also, doing the math: 2GHz / 350k pps = 5714 cycles per packet. 6k cycles to deliver the packet from NIC to one core, then pass it between cores and copy over to application. That ain't bad really.

https://news.ycombinator.com/formatdoc

*Paul Graham

At that point ease of management or package installation probably matters more to developers and it might simply come down to things like driver support and other stability/performance issues where Linux has gotten a LOT of highly-specialized attention from hardware vendors and the HPC world. Back when 10G hardware was just starting to enter the market, we bought cards which came with a Linux driver but it took awhile before that was ported to FreeBSD and longer still before the latency had been optimized as much.

We're not consuming/using the built in network stacks anyway, we're using the OS as a content delivery system. Get us something we can get to the cores on, we're going to pin our applications directly to those cores keeping the kernel relegated to scheduling tasks on whichever NUMA core is further away from the PCIe bus running into the CPU. We'll have the CPU's we're using pegged in a constant spinlock anyway which would make the scheduler think really, really hard about running tasks there. We don't use realtime kernels as it's better for us to pay the price on the occasional spike outlier than to raise the baseline latency up.

Due to my own unfamiliarity, I don't know what BSD's equivalent to isolcpus is. I don't know how to taskset on a BSD. I don't know if the infinibad/ethernet controller's firmware/bypass software works. I don't know how BSD's scheduler works (not that we usually care, but there are times when one can't avoid work needing to be scheduled, things like rpc calls to shut down an app, ssh if you can spare the clock cycles for key verification, etc).

Would dtrace come in handy? Most definitely. Is that enough for us to abandon what we know works? Not yet.

If you need something specific with better performance than that, you should probably look at moving more of the network stack to your application layer.

I remember these claims being made in the late 90s, and perhaps they were true back then, but it's been 15 years, and I would be surprised if Linux hasn't caught up by virtue of its faster development pace, greater mindshare, and increased corporate/datacenter usage.

So, in all seriousness: what recent, well argued essays/papers can you refer me to so I can understand the claim that BSD networking is still better than Linux in 2015?

E.g. on freebsd I recently ran into an ENOBUFS that I've never seen on linux. Although man pages said that they could happen.

Setting aside the fact that a typical fanboi comment is at the top of HN post, are you seriously contending that since one OS does a thing there's no need for other OS's to do the same thing?

Obviously it provides nothing extra, and certainly nothing better, when it comes to a server role. Other than support contracts. And support for 3rd-party apps. And hardware drivers/compatibility. And users that know how to operate it. And brand name recognition. And size of software development community, libraries, tools. And industry reliance on non-portable software (Docker).

Answer to your question: technical superiority is dwarfed by a more popular product.

The JavaScript model of "call some callback when the currently-executing function returns" doesn't work, because C is not event-based. The embedded model of "interrupt whatever's happening and invoke some handler on the current thread of control" is just an absolute nightmare to deal with.

  ; Preparing to listen the incoming connection (passive socket)
  ; int listen(int sockfd, int backlog);
  ; listen(sockfd, 0);

  mov eax, 102		; syscall 102 - socketcall
  mov ebx, 4		; socketcall type (sys_listen 4)

  ; listen arguments
  push 0		; backlog (connections queue size)
  push edx		; socket fd

  mov ecx, esp		; ptr to argument array

  int 0x80		; kernel interruption (this is the "receive packet" call along with MOVs above)

It all comes back to how the kernel can get data back to the caller: https://speakerdeck.com/trent/pyparallel-how-we-removed-the-....

Windows has a distinct advantage thanks to design decisions made in VMS by Cutler et al; namely, I/O via I/O request packets, and much better memory management.

POSIX systems are at a disadvantage because of the readiness-oriented nature of I/O, leaving the caller responsible for "checking back" when something can be done, versus "here, do, this, tell me when it's done". (https://speakerdeck.com/trent/pyparallel-how-we-removed-the-...)

Good luck receiving 50k pps of UDP packets, doing some processing and _sending_ 50k pps responses back. It actually is hard.