Running BSDs on AMD Ryzen 5000 Series – FreeBSD/Linux Benchmarks
phoronix.com
phoronix.com
There's also software like dotnetcore which will just not build. The situation in the desktop is even more problematic. I wish it wasn't so, I like the separation of concerns quite a bit with FreeBSD but it's not something that unfortunately fits my use cases.
Doesn't work on macos or windows either; macos "docker" uses a linux VM.
Personally I prefer jails anyway.
I am not an expert in Windows though; perhaps you are referring to a version of docker that I'm not familiar with which doesn't support Linux services but instead runs Windows services in a native Windows container system? That would be cool and interesting but obviously it wouldn't run most docker images.
Yeah, and the good documentation. Wish more open source projects had these.
We (as in Linux users) used to make fun of Windows for how stupid everything was. You need a GUI for configuration because many options are hidden away in some kind of cryptic registry, deploying services is a PITA etc. Now we have somehow (d)evolved to that same mess. For instance, SystemD and SELinux are such a pain to work with (each for different reasons) that it feels like I have to fight my OS. Wanting an OS that works for me, instead of against me, was one of the major reasons why I switched from Windows to BSD and then Linux back in the 90s.
There's definitely something to be said for *BSD's boring, predictable simplicity. But perhaps there's some kind of force that automatically drives operating systems to bewildering compelxity as their user base grows?
Nobody is forcing SELinux on you, you can turn it off and that's that.
If you're on a Red Hat based system most of policy is there out of the box, providing you with pretty nice additional protection without too much tinkering.
Maybe I'm just getting old and change is starting to grate on me. But systemd does feel like everything-and-the-kitchen-sink.
As for SELinux, sure, you can turn it off. At which point it's useless. The pain begins when you actually want to use it and existing policies aren't enough. It is hugely complex. Tooling (like ausearch and audit2allow) is, erhm, not very good. Want to allow something non standard? Better be prepared to spend a couple of hours. Don't forget to turn your new policy into a module file. And then compile that into a pp file, which is another kind of module? And then don't forget to install that file. I remember administering Windows NT and 2K boxen and really hating how its policies worked, well, SELinux now makes me feel the same.
Both of these tools make a lot assumptions about how things should be done, and they're pushing these assumptions onto the user, whether or not they agree.
And hey, I get it, these are complex domains, and we should be very happy that people are giving us Free (both sense) Software in the first place. But boy, if I could pay €100 to have a sensible operating system, I would. Just not €100 per machine -_-
systemd-as-init is/was not the problem. systemd-as-kitchen-sink is where people started getting annoyed.
It needs journald (which can't ship logs off-host like rsyslog, so you needed to run rsyslog anyway), it pulled in udevd, it's doing DNS, it's doing time syncing, it's doing DHCP/IP and VXLAN(!), it has a boot manager (formerly gummiboot), it's tracking user logins, it's ….
See: https://www.freedesktop.org/software/systemd/man/systemd-jou...
* https://tools.ietf.org/html/rfc5424
* https://en.wikipedia.org/wiki/Syslog
How do I send/forward/import it into my SIEM/ELK/Splunk?
I'm already using syslog for all my networking events as well as Windows logging (via a translator).
Too. Damn. Big.
People don't hate systemd without reason.
There are many cases where it follows some course of behavior that doesn't follow typical user patterns like the principle of least surprise. Example is the dns/resolv.conf issue.
This was a few years ago, so it is very likely that the particular bug that annoyed me was solved since then. Nevertheless, the bug was so unbelievably stupid and it was not only just a coding error but also a problem caused by the wrong architecture of systemd, so I could not accept that it is possible to trust software developers that can make such mistakes.
After that bad experience, I have never touched systemd again.
Even if this was several years ago, systemd had already been in use for several years, so the bug could not be excused by systemd being experimental or something like that.
I use mostly Gentoo Linux, which fortunately does not use systemd, so I had been shielded from its problems. I had heard about some systemd-related controversies, but I had paid no attention to them. I had seen some systemd presentations, which seemed reasonable enough.
One day I was in a hurry, but I had to install Linux on a new computer. Gentoo is nice, but if you do not have a pre-prepared customized installation image, so that you have to install it from scratch, it may need a few hours for compiling everything from sources and for editing the configuration.
So I decided to install Arch Linux, which used systemd. The installation was fast and everything seemed to work OK.
Systemd was always praised for its supposedly fast boot times, but I have seen no improvement in that, compared to my carefully configured Gentoo systems. On the other hand, the computer with systemd was the first computer that I have ever encountered, among many thousands of computers, from the days of Intel 8080 and until the new Zen 3, which was slow at shutdown.
So while the boot time of systemd was OK, the shutdown time was very long, sometimes more than a minute. What is much worse, sometimes the shutdown failed, which is something that I have never experienced before or after. This was on an Intel NUC with a Skylake CPU, so it was not some esoteric, non-standard hardware.
The shutdown failed sometimes due to a race condition, because the systemd daemons exchanged some messages between themselves through dbus, and, IIRC, sometimes it happened that some daemon died before the sender expected and the sender remained stuck in attempts to send to a non-existing recipient, or maybe the dbus daemon died, so there no longer was who to relay the message.
Race conditions may happen in concurrent programming, but to have a shutdown procedure whose success depends on the success of inter-process communication, that shows that everything was ill-conceived at the highest level.
So then I said farewell to systemd, I wiped Arch and I installed Gentoo, which worked perfectly on that hardware, without systemd, but with its much simpler and more reliable openrc.
It's infuriating when your system won't shut down due to some unknown service, which systemd gives [30 seconds remaining] to quit gracefully. OK, you think, I guess it's reasonable to wait and see if it will close cleanly, so you wait. As soon as the timer hits 0, it jumps up again: [90 seconds remaining]. How are you supposed to know how long to wait? What are you supposed to do if it continues forever? All keyboard input is disabled so there is nothing you can do but hard reset your PC and hope systemd unmounted the filesystems before getting stuck.
It's just an unpleasant software.
The big problem is simple interfaces require more effort on the users part which will never happen for a majority of users. And lets face it, people like me, you and most others here are the extreme minority in computing. We aren't afraid of $ vi /etc/config or echo killuser >/n/pcturret/ctl. But writing documentation is boring and "hard" so it never gets done. People start complaining about docs and someone comes up with the bright idea to automate vi /etc/config by replacing it with a poorly thought out contraption that needs binaries and bolted on protocols. That's exactly what dbus is and is best described as duck tape engineering.
Yesterday one of KDE devs posted an article mentioning this sort of phenomenon and linked to an article depicting it in full[1]. And now that you mention Plan 9 it makes me think, because I always dreamt about Plan9 taking over the world and becoming big and everyone using it, but Plan9 is so great because it has never left the 'geek' circle, while others like Linux (and arguably the BSDs, to a lesser extent) already had.
This is why the 9front/cat-v crowd is cantankerous and the humour as dark and cringy as possible. No likes. No stars. No instagram accounts full of pictures some jerk took in the middle of the sahara to show how shallow they are. We only want well thought out code. Otherwise go away.
Too bad it's stuck with hard coded 32 bit dependency. It also cant build on 64 bit systems unless they have 32bit compat (windows/linux win here). There's a fork maintained by some 9front users called Purgatorio. Charles Forsyth also still pokes at it and just added Risc-V support. It's not dead, just resting. Patches welcome.
On the flip side it windows. That OS doesn't hide in the background, rather it screams for attention 24/7, inserting itself in every little task even when specifically told to back off.
The tests appear to compare ZFS and ext4 and clang and gcc as much as FreeBSD and Linux.
Does anyone have some good LWN-type articles that goes deep into the differences between BSD vs Linux internels (also common user land traits in downstream distros)?
https://idea.popcount.org/2017-02-20-epoll-is-fundamentally-...
However, this has always been a weak spot for Linux until recently and it took too many iterations of various partial solutions of the problem to reach the current state.
This was a case when avoiding the NIH approach and looking carefully at the older Windows or FreeBSD solutions would have resulted in an earlier good solution.
I used it under ESXi with physical disks attached as a backup server with hammer/hammer2 recently. Worked great and it was the only FS that offered effective offline dedup.
1. No ECC driver for AMD Ryzen, and no ECC support in memory management.
While you don't need any OS support to use ECC, it's much better to have it.
* Linux has an ECC kernel driver for AMD CPUs that accurately reports the ECC status, this is extremely helpful given the uncertainty of ECC in consumer-grade hardware. On FreeBSD, it's not possible to tell whether ECC is enabled due to the lack of driver. Also, if ECC is not initialized properly by the firmware, the Linux ECC driver will also detect it and try reenabling ECC and memory scrubbing. On FreeBSD, better to check whether your BIOS is broken.
* When an ECC error occurs, a Machine Check Exception is generated by the CPU. On Linux, it will be correctly decoded and recorded. But on FreeBSD, so far there's no decoder, leaving you a mysterious MCA error in dmesg. For example, a correctable DRAM ECC error will be reported as "L3 cache error" (which made many people to falsely believe that Ryzen's ECC was not working on FreeBSD).
* On Linux, if an unrecoverable multibit ECC error is detected in userspace memory, the process will be killed by a SIGBUS (my username checks out?). If the error occurs in kernel memory, it triggers a kernel panic. On FreeBSD, as far as I know, none of the features is supported by the kernel, it increases the risk of data corruption, and it's not a Ryzen or MCE-specific problem. Also, Linux also supports memory offlining, which can be used to temporarily remove bad memory pages from the system after an ECC error, which is not found in FreeBSD.
Fortunately the ECC-enabled flag is easy to check (I plan to make a contribution to the FreeBSD kernel), and a MCE decoder in userspace should also be easy. But implementing the rest is not trivial, the lack of ECC support in memory management is a problem. It's unfortunately that these features are not supported in a server-grade OS like FreeBSD (I guess FreeBSD runs on "real" servers, which implements all of those ECC features in the management firmware, so "real" FreeBSD users don't really need them...)
2. No USB serial system console and netconsole support. On FreeBSD, although tty login via USB serial is supported (just run a getty on it), it seems that there's no way to redirect all dmesg to a serial port (perhaps someone can correct me), FreeBSD's comconsole requires a real physical serial port, not USB, on the other hand Linux's CONFIG_USB_SERIAL_CONSOLE supports USB natively. On Linux, there's also a kernel-space netconsole that redirects all demsg via UDP/IP, this feature is not found in FreeBSD although there was an attempt in 2012.
For USB serial console, I think part of the issue there is that the serial console can interface with the kernel debugger, which might be tricky if it's USB (but I guess USB keyboards probably work in the kernel debugger, so maybe there's a chance?).
Thanks for working on Ryzen ECC. I don't work with a server farm anymore, and I'm too cheap to buy ECC for home use, but someday I might use that support.
On servers with Intel CPUs, FreeBSD warned me promptly, by e-mail, about some corrected ECC errors.
The errors appeared suddenly after many years of use and they kept recurring, pointing to a probable failure in the near future, but being promptly notified, I have replaced the offending DIMM and all went back to normal.
So yes, you are right, for FreeBSD it is better to stick to Intel, unless you have time to change yourself the sources, to add the missing hardware support, which is somewhat easier than attempting to make changes in the Linux kernel.
There are also examples of hardware for which FreeBSD has better support than Linux, but those belong to classes of hardware less likely to be encountered by non-professional users.
For example FreeBSD has better support for magnetic tapes.
This is not just an AMD problem. For example, as I already said, there is no support of unrecoverable ECC errors handling in the FreeBSD memory management subsystem for any CPU, including Intel. On Linux, the kernel has first-class ECC support via the Error Detection And Correction subsystem (EDAC). EDAC can decide to kill a process, to trigger a kernel panic, or even to isolate the faulty memory based on the location of the memory error. On FreeBSD, the only thing that handles unrecoverable ECC errors is the userspace utility sysutils/mcelog [0], the best thing it could do after an unrecoverable error is sending you an email, at which point data corruption may have already occurred [1]. Usually correctable errors occur before unrecoverable ones, so data corruption could be prevented if it's fixed soon enough, but if you get "lucky"...
And unlike writing device drivers, I don't think most people (including myself) is qualified to mess with the memory management code in the kernel. But I have a plan to do something about it, for example, there's nothing stops me from writing a simple script to decode the MCE log and kill the kernel from the userspace. And given how rare unrecoverable ECC errors are, even without graceful handling (like killing userspace process or runtime memory offlining), it should solve my concern of data integrity on FreeBSD.
[0] But corrections are welcomed. I suspect FreeBSD will also halt the kernel on Intel platforms after a fatal MCE due to ECC error, but I didn't find any information and came to a negative conclusion.
[1] I guess it's not really a problem for "real" FreeBSD users, on "real" servers, BIOS can kill the system after an unrecoverable ECC, which probably explains the lack of any unrecoverable error handling in FreeBSD.
So, it depends on what you're doing. A lot of the tests where FreeBSD was much slower were media related; it makes me wonder if some CPU acceleration didn't get enabled; which could be FreeBSD (or Clang) missing a feature, or a benchmark issue or something else.