Their open conect FreeBSD boxes are nuts. The amount of data they spew is crazy. 1/3 of all internet traffic. One service. Mind blowing.
That is because in my case, the data path portion basically talked directly to a couple PCIe boards. It bypassed the entirety of the kernel outside of some setup API's to claim memory/interrupts/etc. That meant the transfer limits generally came down to lack of PCIe or memory bandwidth (depending on which generation of machine/configuration we were using). The CPU's in the machines spent 99.99% of their time running code we wrote. Despite the talents of most OS developers, generic OS/driver code is not optimized for absolute performance in one case, rather it tends to be tuned to perform well over a wide range of situations. The general goal is to be a fair arbitrator of system resources to multiple competing processes. Further, most general purpose OS's are under the assumption that I/O is slow or low bandwidth. Take the entirety of the linux filesystem/block layer/scsi layer, which is written under the assumption that the system is attached to a high latency low bandwidth spinning disk, so burning a few cycles coalescing requests, or handling the page cache isn't a big deal. That code doesn't scale when you plug it into a NVMe disk with 2GB/sec of bandwidth, much less a storage network with 100GB/sec of IO bandwidth.
Anyway, if you throw all these assumptions away and ignore modern "best practices" development models of assembling piles of unrelated libraries to solve a task, you end up with really lean (probably fits in the L1i cache) software that can perform two or three orders of magnitude faster than similar code written using modern methods.
Netflix connections are typically about 1mbit/sec each (older apps open up ~4 connections per video for reasons that are no longer valid but the apps aren't all updated).
So to fill a 100Gbit pipe they have 100,000 connections running at the same time. Which makes filling that pipe super super impressive.
But, there are a bunch of different ways to solve the problems. I guess how impressive it is depends on they have gone about solving their particular cases. There is a fair number of network accelerators that offload individual stream level management to little cores running on the network adapter itself. Cavium, EzChip and now even companies like mellanox are playing in this space https://www.enterprisetech.com/2017/10/04/mellanox-etherneta....
So, i'm not sure the impressive parts are necessarily in the stream counts but what they must be doing to "align" (for lack of a better term) them. AKA the trade offs between keeping a few seconds of a video stream in RAM vs sourcing it from disk/wherever so that multiple users streams are aligned to avoid having to hit a secondary storage medium. In netflix's case I suspect that requiring fairly large buffers on the endpoint allow them to get away with a much lower QoS metric on any given stream.
Put another way, at least the few times I've watched netflix's bandwidth usage, it seems to be bursty. It blasts a few 10's of MB/s of data and then sits idle for a few seconds while the stream plays and then you get another chunk.
They are using either Chelsio or Mellanox cards and they use the offload but they are doing TLS with the Xeon cpus. So they are getting 100Gbit while touching every byte.
And don't under estimate how hard it is to do 100,000 TCP connections. When I was at SGI we had a bunch of big SMP machines (I think they were 12 cpu Challenge) that someone was using to serve up web pages (AOL? It was someone big). Modems brought that machine to its knees. You would think that would be easy but it was not. A single (or small number of) fast streams is easy, a boat load of slow streams is hard. Think about it, if you have a TCP stack that gets a request and then nothing, you have all the overhead of finding that socket, doing that work, then nothing. It's way easier to have a stream of packets all for one socket.
It's that sort of stuff that they worked on so far as I can tell. Your caching idea is nice but the cache hit rate is very very low. They did way more work in the sendfile area, managing the page cache. Did you read Drew's post? It's worth a read for sure.
sendfile() is good, but the general concept tends to waste far to much time doing filesystem traversals, buffer management, dma scatter gather lists, and a bunch of other crap that gets in the way of getting a blob of data from the disk, encrypting it, and passing it off to a send offload to handle breaking up and apply the TCP/IP headers/checksums. Frankly the minimum MSS size is something that ipv6 should have fixed, given that no one is on 9600bps modems, but didn't.
Good for them for realizing that modern machines have a little less than a GB of bandwidth per pcie lane per direction, and memory bandwidth to match. If you don't mess up the CPU side of things you can even touch all that data once or twice and still maintain pretty amazing I/O numbers.
EDIT: Also in the case of x86 NUMA, you _REALLY_ want to make sure that the nvme/source disk, the memory buffer your writing to and the network adapter are on the same node with the core doing the encryption/etc. That is pretty easy if the "application" controls buffer allocation/pooling, but much harder with a general purpose OS which will fragment the memory pools.
It's not fair to Vimeo, because they actually had a copy and YouTube didn't, but because of multiple experiences like that I was surprised to see a positive remark about their quality.
It's the youtube content creators like PewDiPie, Casey Niestat, Kurzgesagt etc that makes Youtube special just like that community that makes Hacker news special.