Serving 100 Gbps from an Open Connect Appliance

Did you get so far as to implement a pageout daemon that scanned vnodes like that? We send only via sendfile, and we give the kernel hints about what should be freed using the SF_NOCACHE flag. This helps a lot. When we think we're serving something cold based on various weighed popularity rankings, we pass the SF_NOCACHE flag to sendfile(). This causes the page to be released immediately when the last mbuf referencing it is freed.

I'm reading through the zfs code and I can see why the kernel is intimidating, all this state you have to gather up to make sense of it. One thing that helps is there are patterns. Just like device drivers, file systems all lock mostly the same way, have a certain pattern. You can blindly follow that and get stuff done. Eventually you have to understand what you are doing but you'd be amazed at how far you can go faking it. That's what I did while I was learning and I did tons of useful work sort of "blind". Eventually stuff comes into focus, the architecture comes first, then the arcane details (usually). And even though I was working in the file system code, there was some stuff (the whole hat_ layer) that I never bothered to learn/memorize, it just worked, I wasn't changing it, shrug. I have a pretty good idea what it was doing at the general level but would have to go learn the details if I wanted to change it.

Kernel hacking is fun and apparently isn't that common a skill any more, people like the comfort of userland. I'm no rocket scientist and I got pretty comfortable in SunOS, IRIX, Sys III, Sys V, etc. Unless you are trying to rewrite the whole thing in a clean room, it's really not that hard. It's hard to know all the details about everything but it is rare that you need to (and even more rare to find someone who knows all that stuff).

If this sort of thing seems interesting, you should grab a kernel and figure out how to build and install it, make a new syscall called im_a_stud() that does some random thing, add it, call it. Off you go :)

Provably optimal classical computing substrate is a hardware cellular automaton. We know this, we've known this for 50 years at least, we still don't go there directly.

The needs of long haul TCP are a bit harder. Check out the TCP RACK RFC and packet pacing to get an idea of how many timers you have firing off. Also HTTP and TLS vs block protos. Netflix numbers are very impressive with all that in light.

Reason I say is that for most smaller operations I'd assume that you'd be more cost effective with dev time to just throw a few more servers at the problem than to go through all the effort to squeeze a few more percentage points of efficiency out of hardware, although I suppose when you're operating at that scale a 1% efficiency gain is enough to justify an entire developer's salary.

Are you considering AMD EPYC processors with their huge number of PCIe lanes?

1 -

"From looking at VTune profiling information, we saw that ISA-L was somehow reading both the source and destination buffers, rather than just writing to the destination buffer."

This part intrigued me, i only have use VTUNE superficially, but how did you profile the destination addresses and correlate it to application level buffers?

2 -

How are was it to profile kernel code, my experience is always hit and miss when using linux/perf-tool.

3 -

A more general question, looking at the optimizations needed to saturate 100G NIC, it seems to me that you are guys are fundamentally approaching the scalability limits of the current IO/Networking model of current OS. How much do you think the current network stack can be squeeze (200G, 300G ?) before a complete redesign is necessary ? Is there any OS out there already design specifically for this kind of IO/Network performance (AIX,solaris maybe ?)

Edit: just noticed some else asked the dpdk one, so feel free to ignore that.

Have you considered moving into userland? E.g DPDK/netmap/etc + BSD stack extracted?