Hacker News Reader

Comment by polskibus | Hacker News Reader

Was FreeBSD your first choice? Or did you try with Linux first? What were the numbers for Linux-based solution, if there was one?

View on HN

FreeBSD was selected at the outset of the Open Connect CDN (~2012 or so).

We did a bake off a few years ago, and FOR THIS WORKLOAD FreeBSD outperformed Linux. I don't want to get into an OS war, that's not productive.

Was licensing also a contributing factor, or was that irrelevant for you?

My understanding is that licensing did factor into the decision. However, I didn't join Netflix until after the decision had been made.

I recall reading that Netflix chose FreeBSD a decade ago due to asynchronous disk IO was (and still is?) broken and/or limited to fixed block offsets. So nginx just works better on FreeBSD versus Linux for serving static files from spinning rust or SSD.

This used to be the case, but with io_uring, Linux has very much non-broken buffered async I/O. (Windows has copied io_uring pretty much verbatim now, but that's a different story.)

Could you expand more on the Windows io_uring bit please?

I have run Debian based Linux my entire life and recently moved circumstantially to Windows. I have no idea how it's kernel model works and I find io_uring exciting.

Wasn't aware of any adoption of io_uring ideas in Windows land, sounds interesting

https://windows-internals.com/i-o-rings-when-one-i-o-operati...

https://docs.microsoft.com/en-us/windows/win32/api/ioringapi...

Windows has had “IO completion ports” since the 1990s which work well and are high performance async for disk/network/other IO operations.

This isn't the same as the old Windows async I/O. ptrwis' links are what I thought of (and it's essentially a 1:1 copy of io_uring, as I understand it).

Its important to consider that we've poured man years into this workload on FreeBSD. Just off the top of my head, we've worked on in house, and/or contributed to or funded, or encouraged vendors to pursue: - async sendfile (so sendfile does not block, and you don't need thread pools or AIO) - RACK and BBR TCP in FreeBSD (for good QoE) - kTLS (so you can keep using sendfile with tls, saves ~60% CPU over reading data into userspace and encrypting there) - Numa - kTLS offload (to save memory bandwidth by moving crypto to the NIC)

Not to mention tons of VM system and scheduler improvements which have been motivated by our workload.

FreeBSD itself has improved tremendously over the last few releases in terms of scalability

a lot of this is in Linux now right ? i am asking a personal opinion and not necessarily a "why-dont-u-move-to-linux" question.

Genuinely curious on where u see state of art when it comes to Linux.

Yes. I ran it for a bake off ~2 years ago. At the time, the code in linux was pretty raw, and I had to fix a bug in their SW kTLS that caused data corruption that was visible to clients. So I worry that it was not in frequent use at the time, though it may be now.

My understanding is that they don't do 0-copy inline ktls, but I could be wrong about that.

Thank you for pushing kTLS!

> FreeBSD itself has improved tremendously over the last few releases in terms of scalability

True. FreeBSD (or its variants) has always been a better performer than Linux in the server segment. Before Linux became popular (mostly due to better hardware support), xBSD servers were famous for their low maintenance and high uptime (and still are). This archived page of NetCraft statistics ( https://web.archive.org/web/20040615000000*/http://uptime.ne... ) provides an interesting glimpse into internet history of how 10+ years back, the top 50 server with the highest uptimes were often xBSD servers, and how Windows and Linux servers slowly replaced xBSD.

(Here's an old HN discussion about a FreeBSD server that ran for 18 years - https://news.ycombinator.com/item?id=10951220 ).

By how much?

Reply on news.ycombinator.com