Hacker News Reader

Comment by ksec | Hacker News Reader

As long as FreeBSD has sustainable interest among cooperate usage it should be fine, but I am not sure if the number of FreeBSD client / cooperate user are growing or shrinking.

WhatsApp moved away from FreeBSD to Linux. And last time there were a few Netflix employees mentioned FreeBSD were used on Open Appliance for historical reason, not technical. So I would not be surprised if someday they move to linux as well. ( Once it offer similar performance )

Mellanox loves FreeBSD, but I am not sure if the same could be said for their new owner Nvidia.

OpenBSD and NetBSD are both an easy sell, one focus on Security and the other on Embedded. I am not sure how to sell FreeBSD, and it needs focus and direction. May be aiming for Network Appliance which is what the majority of its cooperate customer are using it for, where it could offer greater value.

View on HN

Netflix employee here: FreeBSD generally scales well for our workload, and where it doesn't, we can improve it and upstream the fixes without a lot of friction.

Right now, we're serving 200Gb/s of kTLS encrypted video on a single box (see talk at https://youtu.be/8NSzkYSX5nY), and looking at 400Gb/s. I have not heard of others doing this on Linux, and its sexy enough that I assume anybody doing this on Linux would make a splash about it.

In fact, as far as I know, some other big CDN providers that use Linux are just now moving to 100GbE, where as we've been serving at 100Gb/s in production for almost 4 years now on FreeBSD.

Yes! I have been following your crazy work pushing the kLTS limit running into Memory bandwidth problems, and things get wonky once it reaches 75% of max bandwidth. I have always assume FreeBSD's performance was sort of a pride within, so I was surprised another Netflix employees being rather dismissive. ( Edit: I just spend 15min searching for it and couldn't find the comment. I should have favourite that. May be it was my false memory )

Which is one reason why I wish there are CDN running on FreeBSD. So you have an ecosystem / business around FreeBSD ( partly to save guard its future ). AFAIK the only CDN running on FreeBSD is Limelight Network, and they dont really do any pay as you go pricing, everything is ask for quote.

I think not too long ago I floated the idea that Netflix should open a Video Delivery CDN with the Fast.com branding. ( In hindsight it was a silly idea ) And it was for the same reason, more FreeBSD usage = more vested interest = long term sustainability.

Would you recommend a few specific server parts that help you achieve those numbers (in addition to FreeBSD). I am curious about constructing a 100GbE network in my neighbourhood.

We use Mellanox ConnectX-4 NICs (cx5 and cx6 would be fine too), as well as Chelsio T6 in mostly Supermicro boards, making sure to have a full Gen3 x16 (or Gen4 x8).

The Mellanox (and Chelsio) NICs are helpful for us because they support RSS assisted TCP LRO. If you're doing just plain packet forwarding, that would not matter.

Thanks a lot!

I am not sure about the plain packet forwarding or if I am gonna do more, but a few hundred (or thousand) bucks more sound pretty good for future-proofing a neighbourhood level of networking.

Pretty valuable info for a hobbyist looking to get better (and trying to get into community work). Thank you.

Well, the RSS does, but LRO does not.

RSS assisted LRO is a FreeBSD feature where LRO holds batches of hundreds or thousands of packets, then sorts packets by arrival time and LRO hash result. That puts packets from the same connection adjacent to each other, and lets LRO combine them. Delivering packets in batches of hundreds requires some work from the driver (and requires the driver use a new LRO API).

At least on our (Netflix) workload with tens of thousands of active connections, RSS assisted LRO increases our aggregation rate from almost nothing to about 2:1, and saves about 10% CPU.

Its been on my TODO list to add RSS-assisted LRO support to iflib, which would give the intel and broadcom 100g drivers access to it, along with assorted Intel and other 1g, 10g, and 40g devices. But it still hasn't happened yet.

I think he was saying that RSS would still matter for plain packet forwarding.

>I have not heard of others doing this on Linux, and its sexy enough that I assume anybody doing this on Linux would make a splash about it.

You guys have had to do some considerable (from my interpretation of your presentations, at least) work to make this happen even in FreeBSD - do you think that a similar level of effort working in Linux would put it within reach there, too? It seems like there's been a lot of people doing some serious packet pushing with XDP.

> WhatsApp moved away from FreeBSD to Linux.

I'm a former WhatsApp server engineer[1]. WhatsApp primarily moved from bare metal hosting running FreeBSD to Facebook's owned and operating containerized management system which incidentally runs Linux.

We did not make a technical choice to abandon FreeBSD in favor or something else, we made an organizational choice to abandon external hosting in favor of owned and operated hosting which required a lot of technical changes, one of which was switching operating systems.

About half of the server engineering team pre-acquisition was former Yahoo employees, and we had seen what happens when an acquired company does not assimilate the acquirer's basic tech stack --- every issue with hardware or software first has to go through the triage step of is it broken because you're not running the company's OS and the company's package manager and the company's monitoring service and whatever else. That's not a great position to be in when you're trying to run a reliable service, so we decided it would be better to accept Facebook's foundational tech stack and hope the benefits outweighed the costs.

[1] As such, I usually recuse myself from discussions of WhatsApp and Facebook, I'm certainly not authorized to speak on behalf of either; my opinions are my own, etc.

Last time there were Netflix employees claiming Linux can't do more than 60% of FreeBSD's performance in their workloads. Pretty good technical reason if you ask me.

Netflix has modified FreeBSD networking stacks to serve 100 Gbps at edge. But it's just their secret juice.

Nothing is secret. AFAIK, everything is upstream in 13-current

EDIT: The above applies mostly to the kernel. I've run a vanilla FreeBSD kernel on one of our appliances and gotten good performance (and served at 100Gb/s).

In terms of userspace, openssl support for KTLS is only available in OpenSSL's master tree. We'd like to bring them back into the stable OpenSSL that's in FreeBSD's contrib, but we're not sure that's going to happen. And there are some patches to support kTLS with ngninx that are floating around in various forms; I'm pretty sure Mellanox has been giving out a version that should work for FreeBSD as well as Linux ktls.

Same way Google modified Linux to serve whatever they need it to. That’s how Open Source works. (And yet, 60%. Curious.)

Could you please provide a link to those discussions/benchmarks regarding 60% performance? In the presentations by Netflix engineers that I have seen, they've primarily cited licensing as the reason they chose FreeBSD over Linux.

It would certainly be interesting to see a performance comparison between Netflix's open connect appliances with Google's global cache servers.

at FOSDEM the Netflix guy there said they use FreeBSD because of licensing reasons. [1]

This is because they've changed certain parts and don't want to release some changes to the community.

https://archive.fosdem.org/2019/schedule/event/netflix_freeb... (first question covers it @ 44m~)

They should only have to release the changes if they publicly release the modified OS.

Its tricky. For business reasons, Netflix's ISP partners own the Openconnect appliances that are hosted in their data centers. So I think they could request the sources for GPL compliance.

Thank you for posting this Drew.

It's easy to dismiss companies license concerns over the GPL as ignorance, because, they usually are. But this is a good reason that I have never heard of.

Reply on news.ycombinator.com