I've also put together a writeup of how to make fast network servers[1].
Also you should make sure the IRQ of the network card is pinned to one core (that's a system configuration).
But you know, if you really want performance you should go DPDK etc.
Benchmark and simplify is a very good strategy.
Your IRQ observation is good. Advising people to set /proc/irq/IRQ_NUMBER/smp_affinity on Linux is a good idea. I don't know how to do this on OSX though, and I have not found a good programmatic way to discover a good setting. I suspect it might be network-card and network-driver specific, but would welcome suggestions.
Your DPDK observation is also good, however if I want that level of performance, I might get less complexity by threading TCP directly into the network interrupt.
Your example is fantastic -- I realize it's written in C, but perhaps you could explain how in the context of a tool like this (current article) might allow it to function as an end-to-end webserver?
What kind of limitations? What are the advantages, security implications? What is it well-suited for?
Really appreciate a reply -- I'm writing a book on architecture design and I'd really love to include some elements from the bleeding-edge of performance -- you clearly know what you're talking about.
Many marginal increases in complexity (for features, perceived "friendliness", autovivification of configuration, and so on) incur exponential-or-worse increases in latency, exponential-or-worse decreases in throughput, and may be the leading cause of defects (including security) in programming: I have observed the enemy of performance and security is abstraction.
[1]: https://kx.com/
I have noticed every page I scroll causes a comprehensive loss of around 90%, so in reading something that is 10 pagefuls long, I might only be able to produce a tiny part of the program.
Your milage may vary.
I find not scrolling, and just moving my eyes, I rapidly absorb the program, and I find most bugs just by reading the code. This practice is absolutely impossible for me if I have to scroll very far and made difficult by scrolling at all.
I do not know how to do this except to write horizontally.
Some members of that community prefer this style because they find it more readable.
As a side project I implemented a small text editor in this style. I had a working text editor with the ability to load a file, insert and delete text, scroll around a large file, and save changes. It fit within about 100 lines of concise (but I wouldn't say obfuscated) C.
In the end I decided to reformat it and use my usual style before implementing more features. But it was a good experiment and I now understand the appeal of the ultra-condensed style you see in J, K, some Forth, and some Perl code.