These are the slides from my EuroBSDCon presentation. AMA
I'm not the best at network infrastructure, though I'm more familiar with NUMA stuff. So I was trying to figure out how you only got 1-IP address on each box despite 4-ports across 2-nics.
I assume some Linux / Windows devops people are just not as familiar with FreeBSD tools like that!
EDIT: Now that I think of it: maybe a few slides on how link-aggregation across NICs / NUMA could be elaborated upon further? I'm frankly not sure if my personal understanding is correct. I'm imagining how TCP-connections are fragmented into IP-packets, and how those packets may traverse your network, and how they get to which NUMA node... and it seems really more complex to me than your slides indicate? Maybe this subject will take more than just one slide?
I'm afraid that my presentation didn't really have room to dive much into LACP. I briefly said something like the following when giving that slide:
Basically, the LACP link partner (router) hashes traffic consistently across the multiple links in the LACP bundle, using a hash of its choosing (typically an N-tuple, involving IP address and TCP port). Once it has selected a link for that connection, that connection will always land on that link on ingress (unless the LACP bundle changes in terms of links coming and going). We're free to choose whatever egress NIC we want (it does not need to be the same NIC the connection entered on). The issue is that there is no way for us to tell the router to move the TCP connection from one NIC to another (well, there is in theory, but our routers can't do it)
I hope that helps
Wait, I got lost again... You say you can "output on any egress NIC". So all four egress NICs have access to the TLS encryption keys and are cooperating through the FreeBSD kernel to get this information?
Is there some kind of load-balancing you're doing on the machine? Trying to see which NIC has the least amount of traffic and routing to the least utilized NIC?
There's all sorts of subtle details though that's probably just "implementation details" of this system. How and where do worker threads spawn up? Clearly sendfile / kTLS have great synergies, etc. etc.
Its a lot of detail, and an impressive result for sure. I probably don't have the time to study this on my own so of course, I've got lots and lots of questions. This discussion has been very helpful.
------
I think some of the "missing picture" is the interaction of sendfile / kTLS. It makes sense, but just studying these slides have solidified a lot for me as well: https://papers.freebsd.org/2019/EuroBSDCon/shwartsman_gallat...
Adding the NUMA things "on top" of sendfile/kTLS is clearly another issue. The hashing of TCP/Port information into particular links is absolutely important, because of the "Physical location" of the ports matter.
I think I have the gist at this point. But that's a lot of moving parts here. And the whole NUMA-fabric being the bottleneck just ups the complexity of this "simple" TLS stream going on...
EDIT: I guess some other bottleneck exists for Intel/Ampere's chips? There's no NUMA in those. Very curious.
----
Rereading "Disk centric siloing" slides later on actually answer a lot of my questions. I think my mental model was disk-centric siloing and I just didn't realize it. Those slides work exactly how I think this "should" have worked, but it seems like that strategy was shown to be inferior to the strategy talked about in the bulk of this presentation.
Hmmm, so my last "criticism" to this excellent presentation. Maybe an early slide that lays out the strategies you tried? (Disk Siloing. Network-siloing. Software kTLS, and Hardware kTLS offload)?
Just one slide at the beginning saying "I tried many architectures" would remind the audience that many seemingly good solutions exist. Something I personally forgot in this discussion thread.
Are you running anything custom in your Mellanoxes? dpdk stuff
We like to run each server as an independent entity, so we don't run NVMEof.
They're pretty much rock solid.
If you'd like to discuss more, ping me via email. Use my last name at gmail . com.
What an obnoxious, pedantic and naive response.
"And no, you definitely won't be in "much better place" by "ripping it off" because modern NICs have very complex firmware with hundreds (if not thousands) man/years spent implementing and optimizing it."
What do you think I just explained? I literally write this stuff. Modern nic firmware is still written in C and still has bugs. Do you seriously think drivers/firmware aren't going to have bugs? You just said yourself that they're high complexity. I can't believe I'm even needing to explain this. You've clearly never been a network engineer, do you have any idea how many bugs are in Juniper, Cisco, Palo, etc?
If you don't work on bare metal architecture/distributed systems, move along. This isn't sysadmin talk. Almost nothing used is stock, even k8s gets forked due to bugs. You can't resell PAAS with bugs that conflict with customer billing. NFLX isn't reselling bandwidth so they likely don't encounter these issues, they're using something like Cedexis to force their CDN providers at the edge to compete with one another down to the lowest cost/reliability and they're liable for things like this and can be sued/loss based on their contract. They (CDN customers) are acutely aware of when billing doesn't match up with realized BW usage. They'll drop the traffic to your CDN and split more of it over to a "better" CDN until those issues are mitigated - and while you're not getting that traffic you're not getting that customers exoected monthly payment... because now they're buying less bandwith from you because Cedexis tells them that you're less performant/reliable.
ANY large customer buying bandwidth from a CDN does this, none of this is specific to NFLX whom I know nothing about beyond.. "this is how reselling PAAS works."
I bet the next response is "no way NFLX uses a CDN".. LOL. They all do, my friend, HBO, paramount, disney, etc. They aren't in the biz of edge caching.
The real risk is that we introduce a huge blast radius if one of these machines goes down.
If we're only worried about throughput, and everything works out with queueing, there's no need for gen4 NVMes because the storage has more bandwidth than the network. That doesn't mean gen4 is only hype; if my math is right, you need gen4x16 to have enough bandwidth to run a dual 100G ethernet at line rate, and you could use fewer gen4 storage devices if reducing device count were useful. I think for Netflix, they'd like more storage, so given the storage that fits in their systems, there's no need for gen4 storage; gen4 would probably make sense for their 800Gbps prototype though.
In terms of disk I/O, either in the thread or the slides, drewg123 mentioned only about 10% of requests were served from page cache, leaving 90% served from disk, so that would make worst case look something like 45GB/sec (switching to bytes cause that's how storage throughput is usually measured). From previous discussions and presentations, Netflix doesn't do bulk cache updates during peak times, so they won't have a lot of reads at the same time as a lot of writes.
If they're really RAM-bandwidth constrained, then the 24-core 74F3 (which still has all 256MBs of L3 cache) or even 8-core 72F3 may be better.
The more compute clusters the more PCI lanes in EPYC and the SSD lanes go direct per 8cores.
I'd assume TLS is used in large part for privacy reasons (so ISPs can't sell info on what shows are popular)
That proxy would still need to do kTLS to reduce the required memory bandwidth to something the system can manage, and then you're at roughly the same place. The storage nodes would likely still have kTLS capable NICs because those are good 2x100G NICs anyway. It would be easier to NUMA align the load though, so there might be some benefit there. With the right control software, the proxy could pick an origin connection that was NUMA aligned with the client connection on the proxy and the storage on the origin. That's almost definitely not worth doubling the node count for though, even if proxy nodes don't need storage so they're probably significantly less expensive than flash storage nodes.
If what I found is current, Netflix has 1U and 2U appliances, but FB does a 1U switch + groups of 4x 2U servers, so starting at 9U; but I was looking at a 2016 FB pdf that someone uploaded, they may have changed their deployments since then. 2U vs 9U can make a big difference.
However, in my case, and I expect in yours, requirements engineering is the place where you can make the greatest improvements. For example, I can save a few cycles and a few microwatts by sequencing my interrupts optimally or moving some of the algorithm to a look-up table, but if I can, say, establish that an LED indicator flash that might need to be 2x as bright but only lasts for a couple milliseconds every second is as visible as a 500ms LED on/off blink cycle, that's a 100x power savings that I can't hope to reach with micro-optimizations.
What are your application-level teams doing to reduce the data requirements? General-purpose NUMA fabrics are needed to move data in arbitrary ways between disc/memory/NICs, but your needs aren't arbitrary - you basically only require a pipeline from disc to memory to the NIC. Do you, for example, keep the first few seconds of all your content cached in memory, because users usually start at the start of a stream rather than a few minutes in? Alternatively, if 1000 people all start the same episode of Stranger Things within the same minute, can you add queues at the external endpoints or time shift them all together so it only requires one disk read for those thousand users?
It would be fascinating to hear from Netflix on some serious details of the usage patterns they see and particular optimizations that they do for that, but I doubt there's so much they can do given the size of the streams, the 'randomness' of what people watch and when they watch, and for the fact that the linked slides say the servers have 18x2TB NVME drives per-server and 256GB.
I wouldn't be surprised if the Netflix logo opener exists once on disk instead of being the first N seconds of every file though.
What would you rate the relative complexity of working with the NIC offloading vs the more traditional optimizations in the rest of the deck? Have you compared other NIC vendors before or has Mellanox been the go to that's always done what you've needed?
I have video editing skills and have also done some past-time subtitling of videos. I have all of the software necessary to perform both of these tasks and would be willing to do so free of charge.
We did a bake off a few years ago, and FOR THIS WORKLOAD FreeBSD outperformed Linux. I don't want to get into an OS war, that's not productive.
I have run Debian based Linux my entire life and recently moved circumstantially to Windows. I have no idea how it's kernel model works and I find io_uring exciting.
Wasn't aware of any adoption of io_uring ideas in Windows land, sounds interesting
Not to mention tons of VM system and scheduler improvements which have been motivated by our workload.
FreeBSD itself has improved tremendously over the last few releases in terms of scalability
Genuinely curious on where u see state of art when it comes to Linux.
My understanding is that they don't do 0-copy inline ktls, but I could be wrong about that.
True. FreeBSD (or its variants) has always been a better performer than Linux in the server segment. Before Linux became popular (mostly due to better hardware support), xBSD servers were famous for their low maintenance and high uptime (and still are). This archived page of NetCraft statistics ( https://web.archive.org/web/20040615000000*/http://uptime.ne... ) provides an interesting glimpse into internet history of how 10+ years back, the top 50 server with the highest uptimes were often xBSD servers, and how Windows and Linux servers slowly replaced xBSD.
(Here's an old HN discussion about a FreeBSD server that ran for 18 years - https://news.ycombinator.com/item?id=10951220 ).
Not sure if it’s ok to ask.. how many servers like this one does it take to serve the US clients?
But note that these are flash servers; they serve the most popular content we have. We also have "storage" servers with huge numbers of spinning drives that serve the longer tail. They are constrained by spinning rust speeds, and can't serve this fast.
At the time, AMD was the only Gen4 PCIe available, and it was hard to determine if the Mellanox NIC or the AMD PCIe root was the limiting factor. When AMD suggested Relaxed Ordering, that brought its importance to mind.
This is problematic, because sometimes results are not reproducible.
Eg, if I test on the day of a new release of a popular title, we might be serving a lot of it cached from RAM, so that cuts down memory bandwidth requirements and leads to an overly rosy picture of performance. I try to account for this in my testing.
However, how do you test the saturation point when dealing with production traffic? Won't you have to run your resources underprovisioned in order to achieve saturation? Doesn't that degrade the quality of service?
Or are these special non-ISP Netflix Open Connect instances that are specifically meant to be used for saturation testing, with the rest of the load spilling back to EC2?
The biggest impact I have to QoE is when I crash a box, but clients are architected to be resilient against that.
Also, have you looked into offloading more of the processing to an FPGA card instead?
FreeBSD has "async sendfile", which means that it does not block waiting for the data to be read from disk. Rather, the pages that have been allocated to hold the data are staged in the socket buffer and attached to mbufs marked "not ready". When the data arrives, the disk interrupt thread makes a callback which marks the mbufs "ready", and pokes the TCP stack to tell them they are ready to send.
This avoids the need to have many threads parked, waiting on disk io to complete.
I've been doing this for stuff like SMTPE authority servers and ntpd and things that absolutely cannot go down, for over a decade.
There are also VirtIO drivers involved, and according to the article, they had effect too.
Do you have any additional references around this? I'm aware that most rarely used functionality is often broken and therefore usually don't recommend people to use it, but would like to learn about kTLS in particular. I think for Linux OpenSSL 3 now added support for it in userspace. But there's also the kernel components as well as drivers - all of them could have their set of issues.
Well, the printers wouldn't pair with the new APs, certain laptops with fruit logos would intermittently drop connection, and so on.
I probably will never use that brand again, even though they escalated and promised patches quickly - within 6 hours they had found the issue and we're working on fixing it, but the damage to my reputation was already done.
Since then I've always demanded to be able to test any new idea/kit/service for at least a week or two just to see if I can break it.
So, could Linux be tweaked and made as performant for _this_ use case. I expect so. The question to be answered is _why_.
The gist is that the sendfile() call stages the pages waiting to be read in the socket buffer, and marks the mbufs with M_NOTREADY (so they cannot be sent by TCP). When the disk read completes, a sendfile callback happens in the context of the disk ithread. This clears the M_NOTREADY flag and tells TCP they are ready to be sent. See https://www.nginx.com/blog/nginx-and-netflix-contribute-new-...
The overall idea is to copy bytes from disk to the socket with almost no allocation and not blocking, this is the idea right?
2. When serving video, do you use floating point operations at all? Could this workload run on a hypothetical CPU with no floating point units?
3. How many of these hardware platforms do you guys own? 10k?100k?
2) These are just static files, all encoding is done before it hits the CDN.
Rather than DMA from storage to system RAM, and then from system RAM to the NIC, you could conceivably DMA to GPU RAM and then to the NIC for a subset of sends. Not all of the sends, cause of PCIe bandwidth limits. OTOH, DDR5 is coming soon and is supposed to bring double the bandwidth and double the fun.
If so, then yeah, that's a VPN.
Something like close the connection after N requests/N minutes if the nodes are mismatched, but leave it open indefinitely if they match.
There's of course a lot of ways for that to not be very helpful. You'd still have only a 25% of getting a port number that hashes to the right node the next time (assuming tcp port number is involved at all, if it's just src and dest ips then client connections from the same IP would always hash the same, and that's probably a big portion of your clients), and if establishing connections is expensive enough (or clients aren't good at it) then that's a negative. Also if a stream's files don't tend to stay on the same node, then churning connections to get to the right node doesn't help if the next segment is on a different node. I'm sure there are other scenarios too.
I know some other CDN appliance setups do use multiple IPs, so you probably could get more, but it would add administrative stress.
In a past life we broke LAGs up to use different subnets per port to prevent traffic crossing the NUMA bridge.
I’m sure there are good reasons you didn’t take this approach, be interesting to hear them.
Wild, didn't know nVidia was side-eyeing such far-apart but still parallel channels for their ?GPUs?.
Was this all achievable using nVidia's APIs out-of-the-box, or did the firmware/driver require some in-house engineering :)
We keep track of popular titles, and try to cache them in RAM, using the normal page cache LRU mechanism. Other titles are marked with SF_NOCACHE and are discarded from RAM ASAP.
I ask since I was curious why you guys opted not to bypass sendfile(2). I suppose it wouldn't matter in the event that the client is some viewer, as opposed to another internal machine.
Our transport team is working on packet pacing, or really packet spreading, so that any bursts we send are small enough to avoid being dropped by the client, or an intermediary (cable modem, router, etc).
Would these techniques be applicable on arm64 and/or riscv64?