Hacker News Reader

I have essentially the same take. The sysadmin at my company prefers FreeBSD for all these reasons (as such that's what we're running), and he's engaged me a tonne about FreeBSD but all I see is an operating system that's just as good as the other mainstream server Linux distributions. Except now we've got a system that's more difficult to hire for. "Any competent admin can learn it easily" is something I've been told but how many will want to when they could easily go their whole career without encountering it again?

I like your point about Docker vs Jails, I haven't seen it discussed like that before. I keep hearing Jails are more secure than anything else, I'll have to read more into it.

As far as the networking goes, I haven't seen any recent benchmarks to substantiate those claims either. However, considering Netflix uses FreeBSD on their edge nodes and has put a lot of work into the upstream to improve networking (among other things), it wouldn't surprise me if it's technically superior to the Linux stack. Clearly though Linux's networking isn't an issue for most organizations.

And regarding ZFS, ZFS on Linux and FreeBSD's ZFS implementation are now one and the same. It would be nice to see some of the big distributions(or even the Linux kernel) integrate it more directly. This is probably a solid point in favor of FreeBSD, but it's not like it doesn't work in Linux. I'm not a systems guy, so I'm probably out of the loop on this, but Proxmox is the only distribution I've seen with ZFS as a filesystem out of the box, but I don't know how much production use Proxmox sees. I only run it on my home server.

All that to basically say, I like FreeBSD conceptually. I'm just still not convinced that it's doing enough things better to warrant using it over a common Linux distribution for general computing purposes.

I have mostly only seen FreeBSD used for virtual networking devices so I think if that is your speciality (maybe someone who is a devops / networking role - is there a name for that?) you probably encounter it quite a bit. I do not think I would bother learning it or using it for much outside of networking just because it would make hiring competent people harder. Many times you also need to think about what may not be the very best solution from a purely technical perspective but also what is the best from both a people and technical perspective.

One of the things that annoys me to no end ( and I am not sure if this is something specific to Netscalers or FreeBSD networking appliances in general ) is that the damn VM's are the only thing I run that does not properly report memory and CPU usage to VMware. From the hypervisor perspective they are constantly running at 100% CPU (their support has told me that the system reserves the full amount of CPU given but is not actually running that high - and when you actually connect to the box it self reports the number correctly). Not a huge deal but it annoys me to have that little red "X" in the GUI next to the VM name all the time.

That being said I also have not seen any recent FreeBSD vs Linux benchmarks but I would imagine if a company as large as Netflix is already using it en masse then there would need to be not just parity but tangible benefits to swap it all out for some other linux distro. An org starting from scratch of course would be a different beast.

Solid point about networking. No one really seems to dispute its presence there and rightly so it seems (again, I'm a programmer not an admin so my personal experience is limited).

> Many times you also need to think about what may not be the very best solution from a purely technical perspective but also what is the best from both a people and technical perspective.

This is huge when it comes to running the company's software imho. It's useless having the perfect solution if you can't get anyone to run it when "good enough" will have a plethora of folks ready and willing. Especially when it comes to managing the bus factor. When I asked the admin at my org about that, he explicitly said he didn't care about it (that's more an indictment of him than FreeBSD though).

Overall the human factor is something I've been trying to embrace more lately when it comes to work. For personal stuff though I'll be as esoteric as I like.

> the damn VM's are the only thing I run that does not properly report memory and CPU usage to VMware

Well that's just plain frustrating.

I'm seeing a lot about how hiring competent people would be harder. But I think competence is independent of this, so perhaps what you're saying is Linux use masks incompetence? I would really question the chops of somebody who says they can work with Linux but not FreeBSD. People who really know their stuff on Linux would be able to figure it out.

And if you never see FreeBSD again... So what? The stuff you are building on top is probably more relevant than questions like this.

I've played with each of Linux, OSX and BSDs. The commands and syntax are similar enough, it would be a trifle, indeed a matter of pride for any *nix sysadmin to learn another Unix!

And that much should be expected, since they are both in the same Unix family tree (even if Linux was "grafted in").

> perhaps what you're saying is Linux use masks incompetence?

Why do I see so much of this snide needling from FreeBSD proponents? Small dog syndrome?

It matters when you don't want to hire someone to learn a system, and someone who can land a job with their Linux skills does not necessarily want to learn a different system (one that's in a lot less demand). It's a valid concern even if not an insurmountable problem.

The line you are quoting was, for the record, meant to be a tad humorous.

But in regards to your point... I guess what I'm saying is for most cases, the knowledge gap will be very small, and "hiring them to learn" is a huge overstatement.

Within that you will find niches where there's more of a gap. Eg. Practical knowledge of kernel development will not be super transferrable (though some of it will)

> The line you are quoting was, for the record, meant to be a tad humorous.

That's the needling. And I'm sure it's considered absolutely hilarious among the people engaging in it.

But what it comes down to is someone raises a valid concern and you dismiss it and question the competence of others.

You could have said something actually helpful and constructive without being bitter and dismissive, for example that you don't believe it is as big a problem as might be first thought because of the similarities between the systems, and the good FreeBSD documentation available.

Sorry but I see this attitude every time FreeBSD vs Linux topics come up. Not saying all FreeBSD proponents do it or no Linux ones do similar, mind you.

I think it's important to not take oneself too seriously and that is personally why I inject humor of that sort.

The actual point I'm making beyond that, and I think I make it reasonably well if I can say so myself, is that most of your Linux knowledge is transferrable to FreeBSD, and vice versa. It therefore makes me skeptical when people suggest that it isn't so.

> However, considering Netflix uses FreeBSD on their edge nodes and has put a lot of work into the upstream to improve networking (among other things), it wouldn't surprise me if it's technically superior to the Linux stack.

Possible. But it's also possible that the founder of the team was a BSD person. You occasionally get cases where the preferences of one key person affects things in the long run. At this point I'm sure that they're baked in, because removing all that code would not be worth the effort too, so even if it was better when Netflix went online, that doesn't guarantee that it's still true today.

That being said, Linux is also used in a ton of places, including high network jobs. It would take a decent sized amount of evidence to convince me that all of those other places were wrong and basically only Netflix was right to use BSD in network heavy cases.

My suspicion is that either:

1. The difference is minimal to negligent, and not enough to justify mixed OS development

or

2. The difference is significant, but you have to be pushing your network much harder than most teams ever do for it to show up.

Replying to all the post above as well.

FreeBSD networking during WhatsApp and Netflix starting era was definitely better than Linux. Both in their specific usage pattern. So it wasn’t some ideological reason but actual technical one. But that was 10+ years ago.

If I remember correctly from reading Meta / Facebook successfully switch all of WhatsApp large bare metal 4U server to their standardise Linux blade by 2017. They also spent 3 years working on Erlang BEAM and Networking so it was working close enough.

Netflix continues to use FreeBSD and it is working well for them. Both the lead engineer comment and submit update on HN frequently. Do a search and it should be easy to find it. ( on my phone now so can’t quote links )

2022 now I think any general advantage of FreeBSD over Linux in networking performance would likely be minimal. But it also doesn’t make sense to switch to another for the sake of OS unification.

If anyone wants to learn (as opposed to arguing), the papers on Netflix's TLS offloading work are a fun read.

And say what you will, something like 20% of all internet traffic has a FreeBSD endpoint. And doing 400Gb/s of encrypted streaming from one box is quite an accomplishment.

I would argue the reason someone like Netflix or any of the other large orgs using FreeBSD come there is for the simplicity/cohesiveness. If you're looking to do something like in-kernel TLS or something, way easier on something smaller, documented, and with an OS devel team that will likely incorporate your work in future releases.

>And doing 400Gb/s of encrypted streaming from one box is quite an accomplishment.

It is amazing. I keep following their work from 100Gbps, 200Gbps, 400Gbps and now looks like 700Gb/s [1]...... woah

[1] https://news.ycombinator.com/item?id=30061718

About network performance on 2015:

https://github.com/freebsd-net/netperf/blob/master/Documenta...

> It would be nice to see some of the big distributions(or even the Linux kernel) integrate it more directly.

What's holding this back is legal issues, since ZFS's licenses is not compatible with the GPLv2.

> ZFS on Linux and FreeBSD's ZFS implementation are now one and the same.

Don't assume that this means feature parity, not by a long shot. ZFS on FreeBSD affords you native NFSv4 ACLs (hugely important in corporate settings), tight integration with jails, much better delegation support, the FreeBSD boot loader is actually updated to understand all the ZFS features, and offers boot environments and checkpoint rollbacks right from the boot loader menu (and not having to shuffle around with a dedicated /boot partition like you do on Linux).

If none of those features appeal to you, ZFS on Linux will seem very much the same. Indeed, within the limitations of Linux itself, ZFS does very well at providing every feature that Linux lets it provide.

> Is there some conclusive recent benchmark about this. The post uses a 2014 post about ipv6 at Facebook, which I think is far from definitive today. Especially more since it "forgot" to mention that Facebook intended to enhance the "Linux kernel network stack to rival or exceed that of FreeBSD." Did they succeed over these 8 years ?

I would not be surprised if there still were some cases where FreeBSD could do better. The network stack in particular has a vast amount of performance heuristics and tuning where you can just happen to hit the right or wrong side of some performance jump depending on your exact case, for example.

But to go from there to the network stack is more performant is puzzling. There are certainly cases where the Linux network stack is more performant than FreeBSD -- search for linux vs freebsd network performance and you'll find benchmark tests and complaints and anecdotes around the place including freebsd forums and mailing lists where FreeBSD is slower, and yet FreeBSD proponents would (rightly) say these are not proof Linux's network stack is faster and there might be all sorts of reasons for the differences. So you can't have it both ways.

Netflix uses FreeBSD for some things yes. Not sure if number of big internet corporations using an OS would be a favorable metric for FreeBSD either. Again you can't have it both ways. Netflix says FreeBSD is capable of performing well in that kind of environment, it does not say that it outperforms Linux in general or even in that environment. Any more than Google says the opposite.

> Especially more since it "forgot" to mention that Facebook intended to enhance the "Linux kernel network stack to rival or exceed that of FreeBSD." Did they succeed over these 8 years ?

I don't know about Facebook-driven efforts specifically offhand, but I recall attending a talk at LinuxCon 2015 _specifically_ about optimizing the network stack. Given by a Red Hat employee, IIRC.

Facebook has spent a lot of engineering resources on the kernel. Can't speak to their network stack, but if you want to see what they've contributed, they use emails under the @fb.com domain:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

From a quick glance through I see a lot of work on eBPF stuff, memory, I/O subsystem, cgroups, and btrfs.

The systemd-nspawn man page includes the following:

> Like all other systemd-nspawn features, this is not a security feature and provides protection against accidental destructive operations only

Doesn't seem very similar to jails to me.

Exactly. Personal preference is all fine and good, and he's got a right to his own opinions, absolutely.

But to imply that these are empirical differences between FreeBSD and Linux, well, that's nonsense.