Hacker News Reader

Comment by mikece | Hacker News Reader

Is it still accurate that for network bound servers that FreeBSD still outperforms Linux?

Author here.. It depends heavily on the workload. We have man years of optimizations for our CDN workload that we have been upstreaming to FreeBSD. See, for example, the other talk I co-presented regarding Kernel TLS on FreeBSD.

https://people.freebsd.org/~gallatin/talks/euro2019-ktls.pdf

What made Netflix pick FreeBSD over Linux in the first place?

BSDs unwavering commitment to reliability. A Linux kernel crashes. A lot. FreeBSD ran cdrom.com for 365+ days without rebooting. Just saying

What's "a lot" here, and are you sure that's still true? Because my suspicion is that it might have been true before FAANG sunk billons of R&D into Linux, but I would be shocked if Linux kernels cash at any rare worth noticing today. Although it's splitting hairs, really; when is the last time you saw any major kernel crash?

I crash FreeBSD every day :)

you're making me want to start building some BSD boxes

Every year I still see a Linux system or two live-lock. When I was still doing Solaris and FreeBSD admining that never happened.

However most systems aren't pushed to extremes, so these corner cases aren't seen very often, regardless of OS.

Reminder: Facebook tried to run the WhatsApp stack on Linux but they kept having kernel panics

I've got Linux servers that have gone that long without rebooting (and some that are much longer!), but I generally view it as a bad thing and a failing on my part, and would feel the exact same way if I was still running illumos and freebsd servers. Software updates happen for a reason, even on BSD, and servers running for that long mean that they aren't getting those updates.

It was mentioned in the QnA section of a talk posted somewhere in this comment section (probably https://www.youtube.com/watch?v=vcyQBup-Gto).

At least originally, it was because of the license.

Stock, probably not. FreeBSD is a nicer platform to modify for your use case though.

I think it's safe to say any kind of sweeping generalization like this is unlikely to be broadly true (in either direction). Both work quite well.

I think once you get to a certain traffic level you are forced to do kernel-bypass stuff like DPDK. Regardless of Linux or FreeBSD being the kernel.

If it's just serving files you don't necessarily need DPDK/XDP. For server-grade hardware there now is P2P-DMA and TLS accelerators which can offload everything to peripherials while still using normal socket APIs. You get NVMe -(PCIe)-> crypto accelerator -(PCIe)-> ethernet for the bulk of the data.

Neither CPU nor main memory see any of the network packets as long as they stay on the happy path. Only connection setup, DMA orchestration and occasional TLS renegotiation have to be handled.

In the work we've done, the TLS crypto of bulk data is handled entirely in the CPU by the kernel via ktls. Please see the slides, specifically the data flow diagrams.

I think he's talking about the general case (and since XDP is in the discussion, probably the general case for Linux) and not trying to speak specifically to what Netflix is doing for their CDN.

I know Chelsio has crypto directly on the NIC, but are dedicated crypto accelerator cards a thing and are they ever worth it? Why leave the CPU idle when the CPU itself is a good crypto accelerator (AES-NI, ARMv8 crypto)?

AMD Ryzen has a built-in crypto "decelerator" — a FreeBSD driver was written for the crypto engine, but it's disabled by default because it made everything slower than AES-NI. (Though I guess it would be funny to use it to mine bitcoin, since it supports SHA256. AMD — Advanced Mining Devices!)

Intel has a product line called QAT ("QuickAssisT"?) that does crypto acceleration, as well as compression. I don't know how performant it is. There are definitely several older crypto accelerators that were faster than CPUs of the time; I don't know if any of them (outside of QAT) is still relevant.

The AMD Zen1 Crypto Co-Processor is indeed slower than AESNI; I think it's mostly used by stuff like SecureBoot, TPM, etc, and also used internally by the CPU to generate RDRAND/RDSEED data. It was probably never intended to be used by OS drivers and certainly not intended to be any kind of accelerator.

Supposedly QAT is built into the chipsets of Skylake and above now. I've never seen anyone try it, though.

By chipset, you mean northbridge? Or the CPU?

The part I know of that is built into the CPU is a DMA engine called I/OAT; it just does DMA and maybe basic checksum and RAID transformations. It is sometimes confused with QAT (I've personally confused the two...):

https://www.intel.com/content/www/us/en/wireless-network/acc...

The northbridge. my understanding is that they no longer sell the discrete cards to perform these tasks, and instead offload it to chips that come on the boards.

That might be true, but you can still buy the cards from 3rd party resellers, e.g., https://www.newegg.com/p/2AS-006R-00046 .

From what I recall, the chelsio cards only support a mode of encryption suitable for storage devices, and it's not something you'd use for streaming media.

No, not true. The Chelsio card support GCM and CBC crypto in lookaside (like QAT) using the ccr(4) OCF driver, inline ("NIC TLS") with out - of - tree patches, and TLS offload in TOE mode.

Now that ktls is upstream, we are looking at using the ccr crypto acceleration. We've already tested them in inline mode. TOE is not an option for us, since we do innovation in the TCP stack.

Thanks for clearing that up.

Yeah, sorry, I misremembered. The middle device of the chain in the motivating example given for P2P-DMA was for compression, not encryption.

https://www.snia.org/sites/default/files/SDC/2018/presentati...

Netflix is not bypassing the FreeBSD kernel. They bypass userland for bulk data though --- with sendfile, and kTLS, their webserver (nginx) has to negotiate TLS and parse requests and open files, but then it's pretty simple.

Author here.. We're doing ~200Gb/s through the kernel.

That's why I carefully inserted my weasel words of "I think" ;)

Thank you for your contributions. Great information.

Thanks! I often do the same :)

It depends on how exactly each is tuned. Cache and buffer sizes, drivers, TCP congestion control algorithms, and more can affect this hugely, and are different. The defaults on linux (and probably on freebsd) also change reasonably often, so you can expect linux 4.x to have singificant differences from 5.x.

As I understood Linux made a lot of progress in the last couple of years with say zero-copy patches, etc. I thought they reached some kind of parity.

AFAIK Linux is actually more performant these days, network-wise.

Reply on news.ycombinator.com