Hacker News Reader

Comment by patrec | Hacker News Reader

> What also gets lost in the fray is FreeBSD, even with all those Linux-focused tools, is still a compelling and useful operating system

Is FreeBSD compelling and useful at this point beyond as an inferior but more liberally licensed Linux substitute? This is basically what Sony (after their PS2 woes), Nintendo and Netflix (as far as I'm aware; maybe there are technical reasons too) have been using it for. The last clearly competent company I can remember picking FreeBSD for some technical benefits was WhatsApp many many years ago, and as far as I am aware that decision would not make sense today anymore.

View on HN

I used freebsd for my last project, I can’t deny that support for BSD is spotty on cloud providers; but as far as performance and so forth it’s not far behind if at all.

The compelling argument for FreeBSD has always been: zfs, pf and dtrace.

There’s also kqueues which are a much better interface to high throughput I/O than epoll.

The problem is that every time this kind of question is asked the ecosystem in Linux is different. Right now Linux has bpftrace, ZFS (kinda) and IO_Uring; so the gap has closed considerably.

It's historically been a very similar proposition to Solaris: zfs, dtrace, containers, and higher performance.

For as much as I like both FreeBSD and Solaris, Brendan Gregg (mentioned elsewhere in the comments) has a very convincing argument as to why the scales have tipped in favor of Linux in the last few years:

http://www.brendangregg.com/blog/2017-09-05/solaris-to-linux...

ZFS integration was worse in Solaris 10 than in FreeBSD. In Solaris the page cache and ARC was not properly integrated and when using mmap data would be cached twice in both page cache and arc cache and there was weird issues with these contending with each other. I think this was still the case in Solaris 11 also but hard to say since it is closed source and their marketing material was fraudulent. As for the higher performance, I would like to see evidence that Solaris has better performance than Linux.

The one saving grace of Solaris was that it gave you amazing debugging capabilities but the practical outcome of that was just giving you the ability to gain deep insight to see see exactly how bad Solaris was as an operating system. In all my years of using it (10+) I never experienced it as being more performant than Linux and in most cases it was way slower if you did not tune it significantly.

There were some very good ideas that went into Solaris but they were almost invariably poorly implemented.

Solaris regularly beats Linux in performance on the same hardware, and the subsystems which are "slower" aren't really: they just don't lie and trade safety for performance, like Linux does. Linux is fake performance at the cost of correctness of operation.

I would like to see some evidence to back this claim.

The burden of proof is on you, since you're the one clearly advocating for Linux here.

> The burden of proof is on you, since you're the one clearly advocating for Linux here.

Ehh... if you want proof for a specific claim then ask for it. I am asking you for proof of the specific positive claim you made, which is:

> Solaris regularly beats Linux in performance on the same hardware, and the subsystems which are "slower" aren't really: they just don't lie and trade safety for performance, like Linux does. Linux is fake performance at the cost of correctness of operation.

The claims I did make is that in my experience Solaris had to be tuned to be reasonably performant. One example where tuning was needed by default was malloc. The default malloc from Solaris had horrible multi threaded performance . IIRC ZFS also had quite horrible performance by default if not tuned and I recall there were multiple tuning we had to do on new installations specifically relating to mmap to somewhat mitigate the poor integration of ZFS ARC cache and page cache which are documented here: https://github.com/openzfs/zfs/issues/225 and https://lists.freebsd.org/pipermail/freebsd-hackers/2016-Jul... there was also a whole wiki on ZFS tuning for various applications including MySQL.

The Github issue is for (then) ZFS on Linux, not ZFS on Solaris: whoever can read is at a distinct advantage.

The second link is an opinion from some researcher at Pasteur institute in France.

Both are bogus, but thank you for playing.

Solaris hasn't beaten Linux in performance in the past 5 years in anything.

As a matter of fact, you can't point to a single source that proves your point.

"That which is purported without evidence can be disregarded without evidence."

Brendan's "convincing" argument can be summed up as "Linux is better in all areas" and listing some relatively new technologies like BPF but which are horrid to program much less understand, and he lists no technical details. Whatever short-circuit suddenly happened in his head, he can be treated as a hostile witness for all intents and purposes. Him and I had a long one on one discussion and I'm not in the least convinced by his "arguments".

> The compelling argument for FreeBSD has always been: zfs, pf and dtrace.

You forgot to mention the "completeness" of the base system. Where Linux distributions is a "mix and match" of various tools/versions, that may or may not be the same across each of them.

FreeBSD provides a solid base that "just works". There's just one version (per release).

Jails is also another great feature. Linux has cgroups and namespaces, with various tools built on top of it like Docker, LXC and LXD, but (IMO) Jails still beat them in a heartbeat by security and ease of use.

Docker has had more severe vulnerabilities in a single year (2018 IIRC) than Jails has had in it's entire lifetime.

I never saw docker as being built for security, I think it's a common concept that's attributed to them because "contain(ers)" sounds like it contains things, the reality is that they're talking about shipping containers pretty exclusively in the context of docker.

Shipping containers are not secure.

But the content of shipping container #1 does not spill into shipping container #2. If it does, there's a problem.

Similarly with software containers, you don't want data flowing into nearby containers (unless you do want that, in which case you have setup the necessary networking and that should be the only way for the containers in question to share data).

I'm mostly interested in docker et al for the isolation aspect. The "ships the app with all its dependencies" is an interesting byproduct, although all too often it translates it means there's a mess in the amount of dependencies required. And that it's going to be an equal mess come update time -- unless of course the image is never updated in which case there's a chance the image becomes vulnerable (for example, outdated libssl).

> I'm mostly interested in docker et al for the isolation aspect. The "ships the app with all its dependencies" is an interesting byproduct

And that's why i use Jails instead (or LXC if you're on Linux). None of the layers. Any vulnerability is in the "base" image, and easily spotted, and can be hotfixed using normal package tools.

Docker is a mess. Most containers consist of minimum 4 "base images", each of which can contain vulnerabilities. All done to save the "overhead" of installing Python on top of a base Debian image, or whatever your flavor of poison is.

Instead i use Ansible for reproducibility, and scripts for updating/upgrading.

Another advantage to jails is that they're just plain directories in a hosts filesystem, allowing existing vulnerability scanners to scan the entire stack from a single host, instead of using containers that needs to be scanned individually.

> There’s also kqueues which are a much better interface to high throughput I/O than epoll.

Can you clarify why you say this?

One area where kqueue will be more performance is in that it supports multiple updates on interest set in one operation, while epoll does not, which means that doing multiple updates to an interest set will likely be less performant with epoll, however high throughput I/O does not translate to high frequency of updates to an interest set.

I never mentioned performance, I'm suggesting that kqueue is a better 'interface'.

epoll is an interface designed by someone who didn't know what they were doing at the time, that's the only way it can be described.. if you look at alternative implementations (such as IO-Completion ports on Windows) then you can see there's a lot to be desired, one of the things epoll doesn't do, for instance is tell you how many bytes are waiting on a buffer; so you have to read the whole buffer and determine if you're beginning a new unrelated message or not.

There's also the fact that epoll will evict things from the buffer on read; so you can't do a partial read and come back. It's just.. bad. io_uring solves a lot of these issues.

For someone who has good grounding in FreeBSD and the above technologies already, I can see how that decision would make sense (if you know dtrace well, you probably invested several man weeks into it -- why throw that away?). Typically the situation will be the reverse however (almost anyone you can hire at any level will know more Linux than FreeBSD) and the technological benefits you list do not seem like they would make up for that.

ZFS: these days it's developed on and for linux first and foremost, as far as I can tell (FreeBSD moved to ZFS on linux, correct?).

Dtrace: linux finally caught up and basically went one better with perf/ebpf.

kqueues: as you said, io_uring looks like it will take care of that (and probably put linux ahead).

pf: I can see how pf is compelling if what you want is first and foremost some firewall box, and you take whatever OS comes with the firewall (I haven't used pf, but most people seem to agree its significantly nicer than iptables et al).

iptables has been supplanted by nftables; though it's not as highly scalable as pf and uses a lot of iptables terminology.

I begrudge the notion though, that desiring something like pf is because you primarily want a firewall box; the syntax is much cleaner and ultimately the performance is better (without bolt-ons like conntrack).

For example these are basic firewalls with rate limiting

pf: https://git.drk.sc/darkscience/ds-salt/-/blob/master/states/...

iptables: https://git.drk.sc/darkscience/ds-salt/-/blob/master/states/...

here's what we have for nft: https://git.drk.sc/darkscience/playbooks/-/blob/a09507e2a9b6...

I'm not saying you wouldn't desire something like pf when using linux. Of course you would. But in the grand scheme of things, I don't see many people choosing FreeBSD over Linux for pf unless they want a firewall box.

io_uring is a lot more complex than kqueue though.

Netflix folks repeatedly stated that in their benchmarks Linux simply couldn’t push data fast enough.

Could be, could also be they did not understand how to configure Linux, would like to see something more concrete to back such a claim up. Personally I'm skeptical that it is the case.

> could also be they did not understand how to configure Linux

They hired Brendan Gregg [0] so I think they know what's going on.

0 - http://www.brendangregg.com/

AFAIK Netflix has used FreeBSD since 2012[1] and Brendan Gregg only started working at FreeBSD in 2014[2]. So not sure what the point of your statement is.

[1]: https://lists.freebsd.org/pipermail/freebsd-stable/2012-June... [2]: https://www.linkedin.com/in/brendangregg/

Even if that’s true then it remains a failing of the Linux ecosystem that they were unable to understand how to configure it correctly or that the defaults did not work as well as the FreeBSD ones.

But I don’t think it’s the case as they’ve put a lot of work into hiring some of the brightest minds in OS development, including Brendan Gregg

http://www.brendangregg.com/

> Even if that’s true then it remains a failing of the Linux ecosystem that they were unable to understand how to configure it correctly or that the defaults did not work as well as the FreeBSD ones.

Or alternatively they just did not put the time in. No defaults cover all use cases, almost all software I have ever used in high load applications needed some tuning, this includes Solaris, MySQL and MongoDB among others.

> But I don’t think it’s the case as they’ve put a lot of work into hiring some of the brightest minds in OS development, including Brendan Gregg

AFAIK Netflix has used FreeBSD since 2012[1] and Brendan Gregg only started working at FreeBSD in 2014[2]. So not sure this really shows they knew what they were doing when they made the choice to go with FreeBSD. It could be they did but I still would like to something more than just Netflix's choice to back the claim up. Many more companies choose Linux.

[1]: https://lists.freebsd.org/pipermail/freebsd-stable/2012-June...

[2]: https://www.linkedin.com/in/brendangregg/

Interesting, according to this presentation from 2013 there were several reasons to use BSD, but network speed is not among them.

https://people.freebsd.org/~scottl/Netflix-BSDCan-20130515.p...

This presentation also makes no indication that FreeBSD was chosen because it had better performance than Linux: https://youtu.be/KP_bKvXkoC4?t=732

Seems GPL already ruled Linux out.

Don't be. You are talking about the folks that pulled this off: https://people.freebsd.org/~gallatin/talks/euro2019.pdf

7 years or more after choosing FreeBSD. This in no way vouches for their expertise in 2012.

I think that the engineers at Netflix "could figure it out".

As a long time BSD fan (since 1998) who has had to accept the fact that Linux won out in the end, I can vouch that by far the biggest pain point on Linux is its networking stack. Whenever I have to do anything other than set an IP address I throw up in my mouth a little bit. Linux has/had its benefits (packaging, install base), but IPTables or its routing configuration isn't one of them. High performance is certainly achievable, but it has a lot of quirks.

BSD's networking stack has always been more robust. Anybody who's worked heavily with both will tell you that. It did not surprise me one bit that the primary reason they chose FreeBSD over Linux for their CDN was networking. Juniper came to the same conclusion, on top of licensing reasons.

> I think that the engineers at Netflix "could figure it out".

And I would still like to see some evidence.

> Whenever I have to do anything other than set an IP address I throw up in my mouth a little bit.

Very compelling case you make there.

> And I would still like to see some evidence.

There are plenty of talks, papers, etc from both Juniper, Netflix, etc on why they made their choices, but your response is merely "could also be they did not understand how to configure Linux" without any evidence. We're not going to do your googling for you.

> Very compelling case you make there.

https://www.dictionary.com/browse/hyperbole

> There are plenty of talks, papers, etc from both Juniper, Netflix, etc on why they made their choices, but your response is merely "could also be they did not understand how to configure Linux" without any evidence.

Here are the only sources I could find that gave some reasons for Netflix's choice:

- https://people.freebsd.org/~scottl/Netflix-BSDCan-20130515.p...

- https://youtu.be/KP_bKvXkoC4?t=732

Neither of them suggest FreeBSD was faster than Linux in their tests.

From both of those sources I would say avoiding GPL was the primary consideration.

If there is something to back your claim up, link to it. If you cannot link to it then how the heck do you expect me to find it. I'm not the person claiming it exists.

In the presentation they literally list the TCP stack as a technical advantage. In the video the guy literally says kqueue and sendfile as superior freebsd code for their use case.

Also, I never said it's "faster". Most modern OS's on modern hardware can saturate most networking links. I said robust. There's a difference in how much CPU will be uses as well has how stable and consistent it can be.

I'm also not claiming FreeBSD is a magical solution to everything. I've long switched to Linux for various practical reasons, but there are some areas it still shines.

> In the presentation they literally list the TCP stack as a technical advantage.

At the timestamp I shared up to the next slide no comparison is made between Linux and FreeBSD.

And the thread we were in is about someone claiming the reasons why Netflix chose FreeBSD was performance and not licensing. In fact, here is a direct debunk of that claim:

https://youtu.be/vcyQBup-Gto?t=2739

Jonathan Looney, Engineering Manager at Netflix, claiming their primary reason for choosing FreeBSD was GPL avoidance and that subsequent performance testing (which was not the basis for their choice) is not exactly fair because they did not do the same amount of engineering work for Linux as they did for FreeBSD.

> I said robust.

And I said "And I would still like to see some evidence."

... that is more robust than your anecdotal reports of nausea, which I'm not sure really I can translate meaningfully to a measure of robustness.

To which you suggested I should do your work for you like I'm your slave or something. If you can't back claims you make, don't make them, simple.

The telecommunications company I am currently employed for uses FreeBSD for the backhaul applied with CG-NAT. And this is for the simple reason that the Linux stack couldn't keep up with the load.

If FreeBSD performed better out of the box without needing as much or no tuning at all, then it's the better tool for the job, specifically their job.

Maybe, still no evidence to back the speculation up.

They have both Linux and FreeBSD, yet FreeBSD is the one they still use for streaming. A Linux-biased and centric shop like Netflix would ditch FreeBSD the first chance they had if it weren't better at it than Linux.

Both of these things suggest they had many other reasons to chose FreeBSD, and neither suggests FreeBSD was faster than Linux:

- https://people.freebsd.org/~scottl/Netflix-BSDCan-20130515.p...

- https://youtu.be/KP_bKvXkoC4?t=732

Seems GPL was a problem for them.

EDIT: another source suggesting GPL avoidance was the primary reason for choosing FreeBSD over Linux: https://youtu.be/vcyQBup-Gto?t=2731 - and the presenter also says that the performance comparisons they have done between Linux and FreeBSD is not exactly fair because they did not do all the engineering work on Linux that they did on FreeBSD.

https://youtu.be/vcyQBup-Gto?t=2739

Then they shouldn't be going through the kernel at all. FreeBSD will still be slower than accessing the NIC directly via DPDK.

Reply on news.ycombinator.com