Hacker News Reader

Comment by commandersaki | Hacker News Reader

Ran a FreeBSD colocated server for about a decade that went through generations of hardware. I really want to like the OS, except it's most touted feature, the network stack, was consistently unreliable for me using Intel NICs on Supermicro servers. They would go offline usually after some load due to mbuf resource exhaustion. I never got to the bottom of it even though I posted to the bugs database and would diligently follow up and perform experiments. This also happened on different incarnations of server hardware, so it wasn't the same physical NIC having the issue, but different varieties.

Anyways had enough of the random downtime, I just switched to Linux which didn't have these issues.

I'd say the best part of FreeBSD though is freebsd-update which was a game changer from the previous make world shenanigans.

View on HN

Intel has had a somewhat awkward relationship with FreeBSD forever. While they are forced to support it (for $reasons) they never have properly covered it.

I would say as of FreeBSD 12-13 most major issues are addressed from 1gig up to current 100g. There is an odd bug in 2.5g igc where some users have interface stalls whilst others like Netgate are shipping large numbers without issue, waiting to hear if this is firmware or not.

Source: I maintain several of the Intel drivers on a volunteer basis and used to send several Tbit/s to the Internet over them professionally.

Thanks for your contributions!

I colocate all my servers that all run FreeBSD.

Myself has been through generational hardware, and had had zero issues with any apart from when the raid card failed.

Network has been solid. ZFS has just worked. Not sure what your issues were however colocating since FreeBSD 8, and now colocating 16-CURRENT on my the server. FreeBSD has been rock stable in my books.

2x Dell R630 and 1x Cisco U220 M5

    doublerabbit@cookie:~ $ uname -a && uptime
    FreeBSD cookie.server 12.2-BETA1 FreeBSD 12.2-BETA1 r365618 GENERIC  amd64
    10:39PM  up 1752 days,  1:31, 1 user, load averages: 0.64, 1.30, 1.31

I do not believe that your problems are typical for FreeBSD servers.

I have been using FreeBSD servers for around 30 years. Most of them had Intel NICs and I have used at least 5 or 6 different kinds of Supermicro motherboards, both with Xeon and with Epyc.

Most servers have worked 24/7, without being rebooted for years and without having any minute of downtime except when I did some hardware upgrade or kernel upgrade.

I do not doubt that you had the problems described, but there must be some very unusual circumstances that have caused this. I would not be surprised if there was some problem with the version of Supermicro BIOS of your motherboards, and not with FreeBSD, because I have seen many bugs in Supermicro firmware. Or perhaps you had some buggy version of Intel NICs.

There is one advantage of Linux over FreeBSD, which is not widely known. Linux has a huge database of known bugs in various peripheral devices, including Ethernet NICs, and when one of those is recognized it applies workarounds for the bugs.

Like any other operating system, FreeBSD also implements workarounds for peripheral interfaces with known bugs, but because it has a much smaller user base also its database of bugs includes much fewer bugs, typically only those that had been reported by FreeBSD users. Because of this, I have seen cases when some hardware devices did not work well in FreeBSD, while they worked well in Linux, and the reason was always because Linux knew that they must not be used in the standard way, but it applied the corresponding workaround for their bugs.

Only Windows is shielded from the problems caused by bugs, because the hardware vendors write themselves the Windows device drivers and include in them any required workarounds for their bugs.

> There is one advantage of Linux over FreeBSD, which is not widely known. Linux has a huge database of known bugs in various peripheral devices, including Ethernet NICs, and when one of those is recognized it applies workarounds for the bugs.

Almost universal hardware support and workarounds for quirks is precisely the reason why everyone uses Linux even if they might want to use FreeBSD or something else. In other words, this advantage is known and is the reason why Linux is dominant in the server space today.

It is widely known that Linux has "better" hardware support for many peripheral devices, but few know what "better" means, i.e. that a lot of devices that you can buy have various bugs and in most cases their vendors do not document the bugs in any way, because they provide Windows drivers that contain workarounds for the bugs, and they do not care about other operating systems.

The users of other operating systems must discover the bugs and how to handle them by reverse engineering, and here Linux has the advantage of a much greater user base than any alternative, so in most cases the bugs will be identified by some Linux user, and then either the user or the maintainers of the corresponding Linux subsystem will implement a workaround for the bug.

If such a workaround exists, an enterprising *BSD user can look at the code and determine what it is and apply a similar workaround in their BSD.

Usually workarounds/quirks need the highest level understanding of the device driver and the device. There are few people in the world with such mastery and fewer people capable of translating this to another operating system. In concept this is simple but in most cases it is difficult.

This looks to be a theme in FreeBSD, specifically tweaking mbuf numbers because of exhaustion [1], [2], [3], [4] (and not to mention any posted on the freebsd bug tracker). The fact that you have options to tweak mbufs and limits in the first place just doesn't seem like a well thought out design -- in Linux you never have to specify skbuff (Linux's mbuf equivalent) numbers for any reason. I don't even know if my issue was hardware specific or just a FreeBSD-ism to do with mbufs, but it is a class of issue that just doesn't exist in Linux.

[1] https://serverfault.com/questions/335461/pfsense-mbuf-full-w...

[2] https://unix.stackexchange.com/questions/394876/how-to-fix-m...

[3] https://redmine.pfsense.org/issues/5553

[4] https://docs.netgate.com/pfsense/en/latest/hardware/tune.htm...

While you are right that it would be better to never need tweaking the number of buffers, all your links show such a problem caused by the same application, the firewall pf.

I assume that for some reason pf consumes more buffers than other applications, so it is likely to cause such problems. So pf users should preemptively change the default configuration.

I have never used pf, because I prefer the native FreeBSD firewall, both to pf and to the Linux firewall.

With the native firewall and with the many other networking applications that I have run, I have never seen buffer exhaustion, and like I have said, the mean time between reboots was well above a year.

FreeBSD and the other *BSDs have always been famous for not needing reboots, even after years of continuous operation. For this goal, avoiding memory fragmentation caused by dynamic allocation is essential. Thus static allocation for various kinds of resources might be necessary.

I haven't had reliability issues but I could never get decent performance out of the FreeBSD network stack. Across Intel and Mellanox 10 Gbps NIC, it always ended up capping out at around 5 Gbps. Using TrueNAS CORE and OpnSense. Switch to Linux (SCALE and OpenWRT) and I can max out 9 Gbps.

Hail u/cperciva.

Or rather all hail u/cperciva for freebsd-update.

> all hail u/cperciva

Colin Percival (FreeBSD Primary Release Engineering Team Lead) longer uses that ID in Reddit.

Instead: /u/perciva

Most recently, in /r/freebsd https://www.reddit.com/r/freebsd/comments/1rphd7t/freebsd_14...

All hail.

When my company was briefly running FreeBSD on new (5 years ago) Lenovo servers with Intel NICs, we too had some issues. I don't remember the exact details, but our sysadmin at the time had to do a bunch of work to get it fixed.

Does freebsd-update still lack a progress bar?

My internet is slow and I can never tell how far along it is or if it's stalled completely for some reason, which happens a lot.

Reply on news.ycombinator.com