We're migrating many of our servers from Linux to FreeBSD
it-notes.dragas.net
it-notes.dragas.net
* Lack of systemd. Managing services through shell scripts is outdated to me. It feels very hacky, there is no way to specify dependencies, and auto-restarts in case of crashes. Many FreeBSD devs praise launchd, well... systemd is a clone of launchd.
* FreeBSD jail are sub-optimal compared to systemd-nspawn. There are tons of tools to create freebsd jails (manually, ezjail, bastillebsd, etc…) half of them are deprecated. At the end all of your jails end up on the same loopback interface, making it hard to firewall. I couldn't find a way to have one network interface per jail. With Linux, debootstrap + machinectl and you're good to go.
* Lack of security modules (such as SELinux) -- Edit: I should have written "Lack of good security module"
* nftables is way easier to grasp than pf, and as fast as pf, and has atomic reloads.
Maybe I am not tuning Linux appropriately, but I have been in situations where a Linux system is overwhelmed / overloaded and I am unable to ssh to it. I have never had that experience with FreeBSD -- somehow ssh is always quick and responsive even if the OS is out of memory.
Most of the systems that I deal with are Linux, but I still have a few FreeBSD systems around and they are extraordinarily stable.
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1208993
* set disk i/o schedulers to 'bfd' for spinning drives and 'deadline' for solid state, and 'none' for nvme, by creating a file in /etc/udev/rules.d . kernel must have deadline and bfd schedulers compiled in.
* turned on SCSI block multiqueue in kernel config. requires kernel command line option scsi_mod.use_blk_mq=1 to actually enable it. this helped, but did not completely cure the disk i/o problem.
* patched kernel source file ./block/blk-mq-sched.c to hard limit number of queued block device requests to 2, instead of default which is like 32. this absolutely cured the problem. no more disk i/o dragging the system down. doesn't seem to have a major effect on throughput.
* kernel is configured for full preemption, with 1000hz timer frequency.
* for architectures which will boot using the muqss cpu scheduler patch, i enable that with a 100hz timer freq instead.
* overcommit is disabled, as well as swap, and i use earlyoom to ensure process destruction proceeds in a controlled manner in event of memory exhaustion.
That's the bulk of it. No real magic involved; just un-fuck-ifying the default kernel config, which is garbage even for server use IMO.
(This is on a 4.x kernel btw, and I have no plans to downgrade to the 5.x series.)
The fact that these some or all of these tweaks aren't done by default would seem, along with other evidence, to support my belief that Linux is actively being sabotaged by people who don't want it to succeed.
Imagine running a restaurant, normally you can take 32 orders and have the customer sit and wait. One day you get new chefs that can make food 100x faster, but now you can only take one or two orders before the chefs have the food ready and you have to give it to the customer that ordered it. So despite the chefs being 100x faster it now takes much longer to place an order, and the waiting line can grow long with impatient customers.
I think I’ve grown to appreciate it, but not love it. It feels like both systems are at opposite ends of a pendulum.
The RC/unit system was very approachable for me. You could explain it easily, and you could get inside of it and mess around very easily. That affordance/discoverability was awesome.
With systemd, simple things are nicely templated. I can find an example and tweak it to achieve what I want. Complicated things get complicated real fast.
I love not having to track down pid files. I love not having to check if the pid file contents match a valid instance of the expected binary.
The RC system was simple and also full of exceptions and corner cases.
This is important, because this pid file is used to determine which process receives the kill signal. If you get it wrong, and have the right permissions, you can accidentally kill something that you did not intend to kill.
This is further complicated if you want to run multiple instances of Tomcat because now you need to have a unique path for this pid file per tomcat instance.
If the thing that you're trying to run doesn't fork, you then have to execute it in the background and then store the result of $! somewhere so that you know how to kill the process later on.
It's all very error prone and the process for each daemon is often different.
https://tomcat.apache.org/tomcat-8.5-doc/windows-service-how...
The commandline argument is --PidFile and --LogPath. Most, if not all, programs allow you to customise this. It should never be a guessing game, especially when you are the one creating the init file, therefore you are the one in control of the running program.
It's still a guessing game, though, even with CATALINA_PID. It is entirely possible for Java to crash (something which RC scripts do not handle, at all) and another java process starting up which happens to be assigned the same process id as the dead java process. This can not happen with systemd units because each service unit is its own Linux cgroup and it can tell which processes belong to the service.
With respect to servers that crash, approaches differ, but you could use an off the shelf, single purpose daemon respawner, or you could change the whole system, or you could endeavor to not have crashing servers, such that if it crashes, it's worth a human taking a look.
Systemd has heuristics to detect the main process to send the kill signal or detect a crash, but it can guess wrong. Telling it the pid file location makes things reliable.
Plus creation of a pid file serves as a synchronization point that systemd uses to know that the process is ready. The latter is useful when the service does not support systemd native API for state transition notifications.
And the best thing is that even if systemd has to be told about pid files, one never needs to deal with stalled files or pid reuse. Systemd automatically removes them on process shutdown.
I can't stress this enough. Pid files are really bad. The fact that we used to have to use them is a flaw in the OS design. Using them for startup notification is also a hack in and of itself. The kernel has enough information to fully track lifecycle of these processes without writing information into files that are inherently prone to race conditions, and we now have better APIs to expose this information, so we shouldn't need to use these hacks anymore. I don't think there is any Unix-like system left that considers the old forking daemon style to be a good idea, and systemd's handling of pidfiles is really just a legacy compatibility thing.
This still doesn't handle restarting crashed services, and it still is true if you need to make init scripts for your own services outside of the ports tree.
It took me a long time to come to terms with systemd, but I am very glad that I did. For me it makes defining services both easy and reliable.
I looked at one of my FreeBSD servers to see how it was handled in the case of ZNC, an IRC bouncer that I use. ZNC doesn't produce a pid file on startup, so the FreeBSD RC framework tries to find a matching process in the output of 'ps'. [1] As soon as you attempt to run multiple instances of a given service, this falls over completely. [1]
daemon(8) helps -- it handles restarting of processes if they crash, and it can manage pid files and prevent them from getting stale. Nothing on my FreeBSD system uses it. The unbound (dns cache) port uses the pidfile option for rc.subr(8). Looking at how check_pidfile is implemented, it attempts to verify that the process represented by the pidfile matches the process name. Pids also wrap on FreeBSD, so you have a chance of a false positive match if you run multiple instances of a given daemon. I could, of course, change unbound's rc scripts to use daemon, but that feels like a lot of thinking about pidfiles for something that was taken care of a decade ago.
I do like FreeBSD, don't get me wrong, and I use it in my every day life. systemd solves problems for me, and I really like the way it manages process groups by utilizing kernel features.
1. https://docs.freebsd.org/en/articles/rc-scripting/#rcng-daemon
2. https://unix.stackexchange.com/questions/503150/rc-scripts-for-multiple-zope-instances-in-freebsdOkay so you have a PID file somewhere /var/myservice/myservice.pid. The contents of that file is a number which is supposed to correspond to a process you can find in proc.
But PIDs are recycled or more likely your PID files didn't get cleaned up on reboot. So you look at your file and it says 2567, you look up 2567 and see a running process! Done right? Well it just so happenes that a random other process was assigned that PID and your service isn't actually running.
pidfd's are the real real solution to this but the short and long tail of software uses pidfiles.
systemd uses linux cgroups so that it knows exactly which pids belong to the group.
The defaults have surely changed over the years, but pid_max used to be ~32K by default. On the system I'm typing this comment on, /proc/sys/kernel/pid_max is set to 4194304.
The commit which changed the defaults is this one: https://github.com/systemd/systemd/commit/45497f4d3b21230756...
I was focusing on the "surprise startup" scenario where it would all get wiped and reset.
If it blindly reads the pid and sends a kill signal, your odds are pretty good with a limit of 32K pids on a busy system. If it confirms that the process name matches an expected value, you have less of a chance.. but if your process name is bash, java, or python, maybe not as good as you would hope.
I don't have things crashing a lot, but it's naive to pretend that it never happens. It could result in two things: the rc system telling me that everything is fine or the rc system sending signals to some unrelated, poor, unsuspecting, processes.
I don't need a process monitor just for that. I have systemd. :-)
If you're unlucky, though, pid 2567 might match another myservice instance. This can easily happen if you're running many instances of the same service. Even checking /proc/$PID/exe could give you a false positive.
There might still be race conditions but this should cut down the chance dramatically.
Your nice shell-scripts in /etc/rc.d will start EVERYTHING and ANYTHING just in, case, even if you don't always need it (systemd does support socket activation)
Your nice shell-scripts can't handle parallelism (systemd can)
Your nice shell-scripts can't reorder stuff at boot, you have to specify it by hand (systemd can via Needs/RequiredBy etc)
Your nice shell-scripts are worthless if you need to run more than one instance of a service per server (with systemd having many instances of the same service is a feature, not an after-thought)
Your nice shell-scripts won't help you troubleshoot a failing service (systemd will, via systemctl status AND journalctl).
I must say that now it has more features and the configuration format, while could be simpler, is way better than sysvrc magic comments. I'd suggest it should be XML with a simple config file equivalent for simple cases, as the subtree update in YAML-like is a hell anyway.
Different UNIX systems have gone on to replace inetd to address specific shortcomings of the inetd model with launchd, service management facility, systemd, SAM, but the service restart is not an innovation that systemd has brought along. It is a further, fine-grained, improvement over a decade old concept and a tool + ecosystem that has implemented the concept (inetd + rc scripts).
OpenBSD does this stuff right. You keep pushing faulty and buggy crap over and over.
The *BSD rc systems are robust and rely on a framework that abstracts stuff out, you're never going to be digging in there for a pid file's location.
You don't need to dig for the pid file's location, I'll give you that, but it's also possible for this robust rc subsystem to kill the wrong process under the right circumstances.
My system runs unbound (dns cache), so I looked for its pidfile. It's not in /var/run, but rather wherever it happens to be specified in the config. The default is /usr/local/etc/unbound/unbound.pid.
In mainstream Linux distributions, things in /var/run are likely sane if you live in the packaged world. My tomcat example was about adding software outside of the packaged world / ports tree, which is why you would have to track down the pid file to make a service. Given that unbound places its pid file outside of /var/run, which is more chaotic?
I like FreeBSD, and I use it daily! I also very much like the features and organization that systemd brings to the table.
1. https://cgit.freebsd.org/ports/tree/dns/unbound/files/unbound.in?id=6eb036d0d6656a72d27b34557ed4bf1feb1cd4f0A lot of the staying power of Unix is that it's based on a very small set of simple, yet powerful, concepts. RC is one of them.
Systemd allowed to solve them in a reliable way at the distribution level with no need for the system administrator to do anything beyond providing the exports and fstab entries.
It's true that systemd is nowhere near as easy to hack/get into but at the same time I find that the largely consistent definition language/CLI give me a lot less reason to want to do so.
When the need does arise, there's usually a line or two I can add to my service definition to get it to do what I want.
I couldn't say the same about init scripts.
Ubuntu by default sets it to 2GB even if you have something like 16GB RAM. Swap file can then fill up pretty quickly.
Yep, one of my systems got this treatment. It's annoying because OOM has been an issue ever since. I'm playing with EarlyOOM and trying to remember if it'll be a huge pain to resize the partition for more swap space. Thanks for your reply.
Some run their Linux without swap at all, however I think that is better for a server setup where you run a small set of specific set of binaries, thus you know your load.
That is not the case on a desktop where you running a variable set of binaries depending on your task.
I’m somewhat surprised how bad this works on Linux, I don’t think I ever experienced a swap problem on Windows (not even in the 9x days), I guess this is because "Year of the Linux desktop" never happened, Linux is primarily a server and embedded OS, not a desktop OS.
Anything else, 2GB or less or not bother with a swap disc and just use a swap file with a cap at say 2x RAM. Also depends on how much RAM and disc is available.
Security modules do exist, they're different from Linux. Are you sure you're not just expecting FreeBSD-as-Linux?
As for init... What can I say, I've never been anti-systemd, not even remotely, but rc.d is much nicer than sysvinit, and I find it much simpler to understand than systemd. In fact, I think rc.d is an example of how Linux could have alternatively migrated from sysvinit without pissing some people off.
You're right. My original message was wrong, I edited it, while keeping the original content. What I meant is "good security module".
SELinux on CentOS is "enabled by default and forget about it", unless you do something weird. MAC (= Mandatory Access Control) on FreeBSD requires much more configuration. They have some cool stuff like memory limits, but it's not as powerful as SELinux.
A fair bit is included in the base system already, see capsicum for example. Also, see HardenedBSD, which is arguably better than anything Linux has built-in.
No it isn't. There's a reason why government and military servers run hardened Linux with SELinux, and not any of the BSDs.
What would I know, only designed and implemented TS+ systems for a decade!
Microsoft Server systems for government use have been audited and have strict controls for implementation, hardening, securing, etc.
They're probably more secure than BSD.
Not sure what you mean, but the custom Windows Server that the US government uses is likely more secure than BSD.
>Yet I see BSD on the STIG list as well.
No you don't. DISA does not provide STIGs for any of the BSDs. The US government does not use the BSDs for secure systems (TS+ etc.).
As said, they do have STIGs and CIS documents for JunOS but I guess they don't run any Juniper in the US secure networking despite of having certifications.
Now you're strawmaning on top of shifting goal posts again.
>As said, they do have STIGs and CIS documents for JunOS
No they don't. The CIS documents specifically outline the operating systems they support, and there are ZERO BSDs listed (click on Operating Systems)[0].
Similarly, there are zero STIGs for Junos OS. There are STIGs and CIS benchmarks for Juniper network devices, but not for Junos OS. The actual devices could be running on FreeDOS for all we care, but FreeDOS in and of itself would not be allowed to run on any servers. Hilariously, even Juniper is moving away from BSD. Junos OS Evolved is Linux based.
You're fractally wrong.
Security is hard to define, let alone prove. Everyone has a very different definition of security. So first one has to ask, secure from what?
I imagine most of the reason around BSD not on the official list(s) is because it's not as popular. I mean GenodeOS[0] is arguably one of the most secure OS's around these days, but I doubt you can find any public Govt support(by any govt) for running it in production today.
Going back to my original comment, security is complicated, and there is no "secure", but hopefully for a given set of security threats, there is a "secure enough".
The same exists in physical security. Our home door locks are notoriously not secure, but they are generally secure enough for most home needs. But your average home door lock would obviously be idiotic as protection for Fort Knox's gold deposit door.
Comparing BSD to Linux security is complicated, but for most high value targets, the answer probably is, run more than one OS. Root DNS servers and other highly critical internet infrastructure all do this as a matter of common practice. If you are mono-culture Linux only, I worry for your security, as you are effectively a single zero-day away from being owned. Linux, BSD, Windows, etc will all have RCE's and zero-days as a normal part of existing.
0: formal proof secure(sel4), for some definitions of provable even: https://genode.org/
I did not miss that part, you're just mistaken.
>SELinux is not built-in(though it is for certain distributions of Linux).
Wrong. SELinux is 100% "built-in" to Linux. That's like saying btrfs or Wireguard are not "built-in" to Linux because certain distros may or may not have them compiled in. Nonetheless, SELinux is part of the kernel [0].
The rest of your dribble is a painful Gish gallop because you were decisively proven wrong. Mature up a bit and take the L. Fomenting about being proven wrong is against the Guidelines here.
I'm happy to accept that SELinux is now built-in to Linux, the kernel parts do indeed seem to be built in now, news to me, thanks for that. I don't follow Linux kernel stuff much anymore, I haven't contributed to Linux in over a decade.
You seem to assume SELinux is the end-all be all of Linux security. It isn't. I recognize, based on your other comment, that you are fairly new to the field(a whole decade, go you!). Please open your mind and accept differing perspectives, it will do wonders for your ability to reason about security properly.
HardenedBSD[0] essentially implements grsecurity for FreeBSD, plus FreeBSD has built-in capabilities with Capsicum[1], which is true capability based security, which is much different than SELinux's MAC stuff. If you don't believe me, go read the capsicum paper[1] and come to your own conclusions, it might prove enlightening.
Also, see CheriBSD. :)
0: https://hardenedbsd.org/content/easy-feature-comparison 1: https://papers.freebsd.org/2010/rwatson-capsicum/
If you just want to continue hating on me, no reason to respond, we can go our separate ways. If you want to have a reasoned discussion about security, then I'm happy to continue.
Never said or implied anything of the sort.
>I recognize, based on your other comment, that you are fairly new to the field(a whole decade, go you!).
I've been implementing secure, hardened UNIX and Linux probably longer than you've been alive. I just specifically worked on DoD TS+ systems for a decade.
The rest of your Gish gallop is nonsense. Linux also has capability based security ON TOP of all the other aspects of security, SELinux included.
>If you want to have a reasoned discussion about security
That's not possible with you. You instantly showed how little you know about security in general when you flouted your lack of SELinux knowledge, then you proceeded to Gish gallop and sealion because you've been called out.
Stop it.
IMAX-levle projection. The only one here you should feel sorry for is yourself.
> Grow up and interact with humanity better.
"no u". Grow up and take the L with grace.
It really depends on what you mean by "SELinux". The core kernel bits of SELinux, are of course, part of kernel by definition. However, SELinux is not really useful unless it comes with the SELinux policy definition which defines what applications do "normally". This work needs to be done by the Linux distribution, because without it, it's much like relationship between software and hardware. "Without the software, it's just a paperweight."
Not it doesn't. Just like it doesn't "really depend" on what you mean by "btrfs".
> However, SELinux is not really useful unless it comes with the SELinux policy definition which defines what applications do "normally".
"However, btrts is not really useful unless it comes with the userland utilities to actually make and manipulate file systems"
See how pedantic this is? Not only are you being pedantic, you're still wrong, fractally so.
That is exactly operating system does, which is the topic of discussion. Linux OSs such as RHEL as an example.
> This work needs to be done by the Linux distribution
Which are defined as operating systems by CIS and STIG.
This is a poor attempt at "DO YOU MEAN GNU/LINUX?".
Stop it.
Good luck trying to use SELinux without having the policy installed; it's guaranteed to be non-functional. And given that the SELinux policy is distro-speicific, it's not like you can take a random Linux distribution, and enable SELinux and expect it to work. You enable SELinux on the boot command-line, but without the policy installed, it will be dead in the water. And configuring the SELinux policy is extremely non-trivial. It's several orders of magnitude more challenging than running, say, "mkfs.btrfs".
>That is exactly operating system does, which is the topic of discussion. Linux >OSs such as RHEL as an example.
If that's your definition of an OS, then there are plenty of Linux distributions --- aka, an "OS" by your definition --- that do *NOT* have SELinux built in, because they don't have an SELinux policy defined that will work with that distribution's system daemons.
Therefore, by your definition SELinux is not "built in" to all versions of Linux (specifically, "distributions"). Q.E.D.
Wrong again! You can actually have Linux systems that do not support booting from btrfs, but have btrfs-progs installed. Or systems that have neither.
>Good luck trying to use SELinux without having the policy installed
They are installed and included in the Linux operating systems used by the US government, as I stated above. This is a non-point.
> And given that the SELinux policy is distro-speicific, it's not like you can take a random Linux distribution, and enable SELinux and expect it to work.
Wow, it's almost like it be nice if there were standards used by the government and other organizations looking to secure their operating systems. Maybe they can form an agency, I'll call it the Defense Information Systems Agency. They can make standards that secure and lockdown systems, and make sure SELinux is configured properly... We'll call these Security Technical Implementation Guides, STIGs for shor... Oh wait...
> And configuring the SELinux policy is extremely non-trivial. It's several orders of magnitude more challenging than running, say, "mkfs.btrfs".
Immaterial to the matter at hand.
>If that's your definition of an OS
A distribution is an operating system by definition. It's not my definition, it's the definition[0].
>that do NOT have SELinux built in, because they don't have an SELinux policy defined that will work with that distribution's system daemons.
Still "built-in" to Linux. Whether or not it's enabled or complied in is an implementation detail, but it's still "built-in" to Linux operating systems, most definitely those used by the government for secure systems, which was the original point. Thanks for proving my point, Q.E.D.
>Therefore, by your definition SELinux is not "built in" to all versions of Linux
Using your illogic, there are BSDs that send your password through plaintext over the wire because they only have rlogin. They don't have SSH "built-in".
SELinux is absolutely "built-in" to the Linux kernel and operating systems. Whether or not specific implementations of Linux have it compiled and enabled is besides the fact that it is built-in, not third-party.
You're fractally wrong again. Take the L and stop while you're this far behind.
They're probably more secure than BSD.
>SELinux looks nice on paper - another box to check - but it’s just another mitigation later, not something that can be considered “trusted”.
This is fractally wrong.
# PROVIDE: mumbled oldmumble
# REQUIRE: DAEMON cleanvar frotz
# BEFORE: LOGIN
# KEYWORD: nojail shutdown
> The `REQUIRE' keyword is misleading: It does not describe which daemons have to be running before a script will be started.
> It describes which scripts must be placed before it in the dependency ordering. For example, if your script has a `REQUIRE' on `sshd', it means the script must be placed after the `sshd' script in the dependency ordering, not necessarily that it requires sshd to be started or enabled.
Your systemd unit file is backed by pages and pages of docs that must be comprehended to understand and hack on. Unix developers have all they need from the script. Furthermore, its all in context of existing unix concepts and thus your unix experiance is paying dividends.
What I'm saying is that systemds documentation currency (if you accept my metaphor) is spent on covering its accidental complexity and it's voluminous. If you disagree with me, that's fine. This is just my experience as a linux user that's had to deal with systemd.
If your claim is that systemd man pages are well written documentation then I think you're exaggerating and I'll wager you've relied on stackoverflow examples or tutorial blogs to solve your systemd issues--because I have. The reason for this is because the number of concepts and abstractions that you have to piece together to solve your problem is massive. But yeah, it's just a 5 line Unit file. I prefer strawberry kool-aid, thanks.
And about the voluminous documentation, well man pages are supposed to be comprehensive and cover every single aspect of the tools being described. They're not there to just be an intro to systemd for new users and administrators. If you want something like that, look no further than the "systemd for Administrators" series of articles written by the systemd author himself. https://github.com/shibumi/systemd-for-administrators/blob/m....
It't not the unit file that's the problem, it's the mountains of junk, low quality C code written by an obnoxious, arrogant twit named "Linux Puttering" who has proven for 15+ years he couldn't care less about code quality or system reliability.
Besides the anecdotes shared by others over the years about the horrible experiences they've had with systemd, I have one of my own to share. When developing my own distro to escape the bloated, laggy hell that is Ubuntu, I started the build on my existing Ubuntu system. I found out the hard way that accidentally double mounting virtual filesystems on the target volume causes systemd to crash the system after about 60 seconds, with no possible way to recover. On MY system, with no junky ass systemd, making this error harms nothing at all and can be easily fixed.
The people who talk about "buggy, hacky" shell scripts appear to be some of the same type of people who shrink in horror from the idea of compiling their own kernel, or working at the command line. (i.e. not really "hackers" at all.) There is nothing at all wrong with using shell scripts for startup. It is in fact the simplest, and IMO most elegant way of doing the job, and no it isn't buggy or hacky in the least. The file system is the database and unit file and the already existing shell is the interpreter.
My system starts much more quickly than Ubuntu and is much faster and more responsive in daily use also, so the "startup time" excuse is a myth, and practically all of the other contrived examples people use to justify the use of systemd can be done BETTER using shell scripts in conjunction with small, light weight, single purpose utilities built the UNIX WAY.
You could say that a shell script is a config file for bash, and you have to understand bash to understand what it's doing. But a shell is both simpler than systemd, and something that anyone working with Linux already understands.
Try building and maintaining a linux distro without systemd, especially for a large organization that needs to write their own init scripts. And especially when a large number of the devs in that org aren't shell experts, or don't understand the difference between /bin/sh and /bin/bash. And so on.
Here's another example: https://lwn.net/Articles/701549/ before systemd, for complex NFS setups, the sysadmin _had_ to write the init scripts per-site or per-machine. With the solution in the article (systemd generators) one set of unit files shipped by the distro solves the problem for over 99% of users, including most of the aforementioned complex setups.
I think it actually shows a problem, which is that BSD is designed for all your machines to be special snowflakes with individual names, edited config files, etc instead of being mass managed declaratively. So you need to know how to do everything because you’re the one doing it.
Why is there a chapter on custom kernels under "common tasks" that assumes you're going to have a C compiler and kernel source on your machine and want to installkernel on that same machine?
As for 500 lines - take a look at proper rc scripts, eg the ones in FreeBSD. They are mostly declarative; it’s nothing like Linux’ sysv scripts, which were in some ways already obsolete when first introduced (runlevels? In ‘90s, seriously?)
The amount of effort systemd went through to make existing software work is genuinely heroic.
This is the exact opposite of a good thing.
Why is unlimited customization a good thing in the context of a system init?
I mean the shell is still used everywhere, like e.g. to configure and control systemd.
Still I would say a lot of core shell tools are indeed outdated (for backwards compatibility).
If I understand correctly, the issue is mutex on file objects at the kernel level. Basically it is a failing of the implementation of the "file" abstraction. Or perhaps a failing of the "file" abstraction itself?
Per Wikipedia [0] (because I had no idea what a TOCTTOU race condition was)
In the context of file system TOCTOU race conditions, the fundamental challenge is ensuring that the file system cannot be changed between two system calls. In 2004, an impossibility result was published, showing that there was no portable, deterministic technique for avoiding TOCTOU race conditions.[9]
Since this impossibility result, libraries for tracking file descriptors and ensuring correctness have been proposed by researchers.
I was curious to see how a different abstraction would avoid this problem. Plan 9 FS appears to have had this issue at one point [1], but notably not because of the FS implementation itself. Rather the problem was caused by the underlying system's executing outside of the P9FS ordered access to a given file (someone please tell me if my understanding is incorrect).
Here's an article that talks about the problems of this (and other) race conditions from the level of abstraction [2].
NOTE: I'm not claiming that P9FS is immune from this sort of attack, I'm only commenting on what I've found.
[0] https://en.wikipedia.org/wiki/Time-of-check_to_time-of-use
[1] https://bugs.launchpad.net/qemu/+bug/1911666
[2] https://gavinhoward.com/2020/02/computing-is-broken-and-how-...
I'm not sure initv "worked okay".
You may want to look at vnet, which gives jails their own networking stack; then you can give interfaces to the jail. If you use ipfw instead of pf, jail id/name is a matchable attribute on firewall rules; although it's not perfect, IIRC I couldn't get incoming SYNs to match by jail id, but you can match the rest of the packets for the connection. And that brings up the three firewalls of FreeBSD debate; maybe you had already picked pf because it met a need you couldn't (easily) meet with ipfw; you can run both simultaneously, but I wouldn't recommend it. Nobody seems to run ipf, though.
Edit: you may also just want to limit each jail to a specific IP address, and then it's easy to firewall.
My biggest annoyance, apart from the obvious lack of systemd, was a social one: Any time I had to look up how to accomplish some non-trivial task I would inevitably find a thread on the FreeBSD forums by someone else who has had exactly the same problem in the past, together with a response by some dev along the lines of "why would anyone ever need to do that?".
This is the principal reason why I've leaned toward Debian stable (which also aims at being a stable UNIX like FreeBSD) for decades, even though Debian has also been infected with systemd and has made other questionable decisions. Alternatively, I've also had good luck with Void, Alpine, and Artix. (I've had difficulties with electron apps on Void desktop, but it runs flawlessly on servers.)
either that or "don't do an upgrade, build new ones and replace the old ones instead"
If by any chance it was related to sound, today is your lucky day as I just solved that this morning before work.
Ansible, SaltStack, all the devops automation stuff is available and can certainly handle "freebsd-update fetch; freebsd-update install; reboot".
That was your mistake. With BSDs, you use the man pages ;-)
I don't think this has anything to do with graduating from berkely, I'm a hobo with a keyboard.
No they are not. Systemd has a way bigger scope then launchd. It's like saying a truck is the same thing as a family car because they both solve the mobility problem.
> FreeBSD jail are sub-optimal compared to systemd-nspawn.
systemd-nspawn isn't a container. For example it doesn't manage resources such as CPU or IO. Again, the scope is way different and in this case there is a whole slew of things systemd-nspawn isn't going to manage for you.
BTW launchd doesn't have a feature like 'systemd-nspawn'.
> Lack of security modules (such as SELinux) -- Edit: I should have written "Lack of good security module"
And how is SELinux a good security module? SELinux with it's design fell in the same pitfall dozens of security systems did before it; ACL-Hell, Role-hell and now with SELinux we also have Label-hell.
It does[1]. At the end systemd-nspawn runs in a unit, which can be (like all systemd unit) resource-controlled.
> BTW launchd doesn't have a feature like 'systemd-nspawn'.
Neither does systemd. systemd-nspawn is just another binary, like git ant git-annex. The only difference from git and git-annex is that systemd and systemd-nspawn are maintained by the same team.
But like git and git-annex, most systemd installations don't have systemd-nspawn, but systemd-nspawn needs systemd.
> And how is SELinux a good security module? SELinux with it's design fell in the same pitfall dozens of security systems did before it; ACL-Hell, Role-hell and now with SELinux we also have Label-hell.
SELinux is generic enough to allow for sandboxing of services. But in its mainstream use, SELinux is well design that it allows for reuse of policies. Most people don't care about SELinux, they just run CentOS, install RPM from the repository, and everything works out of the box with heighten security. (= if there is a vulnerability in any of these packages, the radius-blast of the attack is limited thanks to SELinux's Mandatory-Access-Control)
[1] https://www.freedesktop.org/software/systemd/man/systemd.res...
That's not, strictly speaking, true. systemd-nspawn is able to do the cgroups-based containers fine without systemd as init, though you do lack some of the fancier resource control and networking features.
I hear it's taken over dns. Has it taken over sound too?
When will it be able to read mail?
systemd the family of tools has a DNS server, systemd-resolved. Like all other tools in the family, using the init system does not require using the other tools, and sometimes also vice versa.
In the broader "Poetteringware" family of tools, sound is handled by pulseaudio, though the server side of pulseaudio is in the process of being replaced by pipewire.
And btw is that why systemd using distros lock up for minutes on boot if the machine doesn't have internet connectivity?
False.
> And how is SELinux a good security module?
It's good enough for most environments requiring RBAC. Additionally, it's complemented by the excellent sandboxing ability of SystemD unit files.
This, this and this again !
After decades of cron, I discovered systemd timers via a passing comment I read here on HN.
My god is it amazing. No more hacky work-arounds in my scripts, systemd now takes care of all the magic such as random timings etc.
I'll never go back to cron.
For example, you can schedule a timer to run 10 minutes after boot. Or a timer that actives 10 minutes after it has last finished running (note: not when it started last time but when it finished! So if the proc takes 10 hours, there is a 10 minute gap between runs. If it takes 10 minutes, there is still a 10 minute gap).
You can also schedule something 10 minutes after a user logs in (or 10 seconds later, etc.).
Additionally you get Accuracy and RandomizedDelay. The former lets you configure how accurate the timer needs to be, down to 1 sec or up to a day. So your unit now runs somewhere on the day it's supposed to run. And with the later you can ensure that there is no predictable runtime, this can be important for monitoring.
My biggest favorite is Persistent=. If true, systemd will check when the service last ran. And if it should have been scheduled atleast once since then, it'll activate. I use this for backing up my home PC. When I do a quick restart, no backups are done but when I shutdown for the night, first thing in the morning my PC has a backup done.
It will also run things that were scheduled to run when the server was turned off.
If it ran every hour, all servers would either hit the backup store at the same time or you would have to manually disperse them. Randomized Delay is great if you want to avoid this problem with short running jobs but it doesn't work well when most jobs take 10 minutes or more and the delay becomes larger than the timer's repeat interval.
So in that case, using a "run every 60 minutes" scheme is a massive advantage that reduces coordination needs.
http://fcron.free.fr/doc/en/fcrontab.5.html
1.0 released 21 years ago.
I'm not a fan of systemd personally but I do understand it has some good parts to it (such as the ones you've listed). That all said, you can still specify dependencies in FreeBSD with the existing init daemon. Albeit it's a little hacky compared with systemd (comments at the top of the script). But it does work.
> FreeBSD jail are sub-optimal compared to systemd-nspawn. There are tons of tools to create freebsd jails (manually, ezjail, bastillebsd, etc…) half of them are deprecated. At the end all of your jails end up on the same loopback interface, making it hard to firewall. I couldn't find a way to have one network interface per jail. With Linux, debootstrap + machinectl and you're good to go.
It's definitely possible to have one interface per jail, I've done exactly that. However back when I last did it (5+ years ago?) you needed to manually edit the networking config and jail config to do it. There might be a more "linuxy" way to do this with Jails now though but its definitely possible. eg https://etherealwake.com/2021/08/freebsd-jail-networking/
Serverless is killing containers. Its killing the need to care about if its freebsd or linux, does it run my api fast enough?
sort of. Systemd took a lot of lessons from launchd but also from sysv and upstart. For anyone who hasn't read Lennart Poettering's "Rethinking PID 1" post[1] I highly recommend it. You'll understand the history, but most importantly you'll understand systemd a ton better.
I think it was a ZFS dev who complained he had to update 150 files to port ZFS to systemd. Simples?
And you have to love all them binary log files, with their dynamic and spontaneus api. Yes, everything is always new and exciting with systemD.
I use devuan. A debian fork with a choice of init systems (well, everything but systemD. ;) It took the dev/devs 2 years just to swap out/revert the init system.
Also, just learned devuan's website is JS and cookie free. https://www.devuan.org/
Edit: I will concede that systemD is great for a lot of people. I honestly wish them success. The work that goes in to building, maintaining, and using it, is substantial. It must be a boon for the economy and job creation.
I suppose that the end goal of the systemd project is an ability to deploy a production Linux system with just systemd and busybox, and run all software from containers.
Not that it's a bad thing to strive for. But it's not going to be Unix as we know it.
Right, that's Plan9.
(Yes, I do regret that Linux was a hyper-succesful project that replicated Unix, not Plan9. But Plan9 is more complex, maybe it was just infeasible.)
To be more specific is it a replacement of more and more Linux userland that nobody asked. I would be ok if systemd would be a process management system that replaces init with a standardised way of managing services (it would be amazing if it was written a safe language, if it was respecting configuration files, if it did not take over managing system limits, etc. etc.).
Unfortunately it is trying to do too much.
(I'm still able to run Void Linux as my primary desktop without systemd.)
Seriously, though, read the article. The 1.2M lines includes removed lines from refactoring. So, it is not completely made up, as implied in that comment. https://www.phoronix.com/scan.php?page=news_item&px=systemd-...
https://twitter.com/pid_eins/status/1214268577509003266
https://www.phoronix.com/misc/systemd-eoy2019/files.html
Including the number of lines in static tables and documentation hardly seems like a meaningful comparison.
I suppose you didn't use vnet? It's a vastly better jail networking experience. You can pretend jails are separate machines, connected via ethernet. I don't think anyone who knows about vnet chooses not to use it!
> I couldn't find a way to have one network interface per jail.
I think vnet is what you want.
> nftables is way easier to grasp than pf, and as fast as pf, and has atomic reloads.
pfctl allows you to do atomic reloads. `pfctl -f`
FreeBSD jails has quite a bit of history (developed in 1998) and made public in the year 2000. FreeBSD has had VNET for use with jails allowing you to add epair interfaces connected to a bridge, or add a physical interface or VLAN to a jail. This feature has been in FreeBSD since 8.0 (2009) and enabled by default since 2018. It also allows you to run PF in each jail.
Sadly many people a bitten by outdated forum and blog posts when it comes to jails.
I agree that firewalling jails with loopbacks is a pain, but most people don't do it that way anymore.
Also PF has always had atomic reloads.
Again, mature, simple and secure, with very few surprises lurking. There's likely fewer lines of code to support all jail functionality than in systemd.
> The whole system is managed by the same team
Mostly philosophical.
> FreeBSD development is less driven by commercial interests.
Mostly philosophical.
> Linux has Docker but FreeBSD has jails!
IMO, this comparison is a mistake. In the Linux world, systemd's nspawn is very similar to Jails. It's a glorified chroot, with security and resource management. All the systemd tools work seemlessly with nspawn machines (e.g. `systemctl status`). Containers à la Docker are a different thing.
BTW, I thought the last sentence about security issues with Docker images was strange. If you care about unmaintained images, build them yourself. On the other side, the FreeBSD official documentation about Jails has a big warning that starts with "Important: the official Jails are a powerful tool, but they are not a security panacea."
> Linux has no official support for zfs and such
Fair point, though I've heard about production systems with zfs on Linux.
> The FreeBSD boot procedure is better than grub.
YMMV
> FreeBSD's network is more performant.
Is there some conclusive recent benchmark about this. The post uses a 2014 post about ipv6 at Facebook, which I think is far from definitive today. Especially more since it "forgot" to mention that Facebook intended to enhance the "Linux kernel network stack to rival or exceed that of FreeBSD." Did they succeed over these 8 years ?
> Straightforward system performance analysis
The point is not about the quality of the tools, but the way each distribution packages them. Seems very very low impact to me.
> FreeBSD's Bhyve against Linux's KVM
The author reluctantly admits that KVM is more mature.
I like your point about Docker vs Jails, I haven't seen it discussed like that before. I keep hearing Jails are more secure than anything else, I'll have to read more into it.
As far as the networking goes, I haven't seen any recent benchmarks to substantiate those claims either. However, considering Netflix uses FreeBSD on their edge nodes and has put a lot of work into the upstream to improve networking (among other things), it wouldn't surprise me if it's technically superior to the Linux stack. Clearly though Linux's networking isn't an issue for most organizations.
And regarding ZFS, ZFS on Linux and FreeBSD's ZFS implementation are now one and the same. It would be nice to see some of the big distributions(or even the Linux kernel) integrate it more directly. This is probably a solid point in favor of FreeBSD, but it's not like it doesn't work in Linux. I'm not a systems guy, so I'm probably out of the loop on this, but Proxmox is the only distribution I've seen with ZFS as a filesystem out of the box, but I don't know how much production use Proxmox sees. I only run it on my home server.
All that to basically say, I like FreeBSD conceptually. I'm just still not convinced that it's doing enough things better to warrant using it over a common Linux distribution for general computing purposes.
One of the things that annoys me to no end ( and I am not sure if this is something specific to Netscalers or FreeBSD networking appliances in general ) is that the damn VM's are the only thing I run that does not properly report memory and CPU usage to VMware. From the hypervisor perspective they are constantly running at 100% CPU (their support has told me that the system reserves the full amount of CPU given but is not actually running that high - and when you actually connect to the box it self reports the number correctly). Not a huge deal but it annoys me to have that little red "X" in the GUI next to the VM name all the time.
That being said I also have not seen any recent FreeBSD vs Linux benchmarks but I would imagine if a company as large as Netflix is already using it en masse then there would need to be not just parity but tangible benefits to swap it all out for some other linux distro. An org starting from scratch of course would be a different beast.
> Many times you also need to think about what may not be the very best solution from a purely technical perspective but also what is the best from both a people and technical perspective.
This is huge when it comes to running the company's software imho. It's useless having the perfect solution if you can't get anyone to run it when "good enough" will have a plethora of folks ready and willing. Especially when it comes to managing the bus factor. When I asked the admin at my org about that, he explicitly said he didn't care about it (that's more an indictment of him than FreeBSD though).
Overall the human factor is something I've been trying to embrace more lately when it comes to work. For personal stuff though I'll be as esoteric as I like.
> the damn VM's are the only thing I run that does not properly report memory and CPU usage to VMware
Well that's just plain frustrating.
And if you never see FreeBSD again... So what? The stuff you are building on top is probably more relevant than questions like this.
And that much should be expected, since they are both in the same Unix family tree (even if Linux was "grafted in").
Why do I see so much of this snide needling from FreeBSD proponents? Small dog syndrome?
It matters when you don't want to hire someone to learn a system, and someone who can land a job with their Linux skills does not necessarily want to learn a different system (one that's in a lot less demand). It's a valid concern even if not an insurmountable problem.
But in regards to your point... I guess what I'm saying is for most cases, the knowledge gap will be very small, and "hiring them to learn" is a huge overstatement.
Within that you will find niches where there's more of a gap. Eg. Practical knowledge of kernel development will not be super transferrable (though some of it will)
That's the needling. And I'm sure it's considered absolutely hilarious among the people engaging in it.
But what it comes down to is someone raises a valid concern and you dismiss it and question the competence of others.
You could have said something actually helpful and constructive without being bitter and dismissive, for example that you don't believe it is as big a problem as might be first thought because of the similarities between the systems, and the good FreeBSD documentation available.
Sorry but I see this attitude every time FreeBSD vs Linux topics come up. Not saying all FreeBSD proponents do it or no Linux ones do similar, mind you.
The actual point I'm making beyond that, and I think I make it reasonably well if I can say so myself, is that most of your Linux knowledge is transferrable to FreeBSD, and vice versa. It therefore makes me skeptical when people suggest that it isn't so.
Possible. But it's also possible that the founder of the team was a BSD person. You occasionally get cases where the preferences of one key person affects things in the long run. At this point I'm sure that they're baked in, because removing all that code would not be worth the effort too, so even if it was better when Netflix went online, that doesn't guarantee that it's still true today.
That being said, Linux is also used in a ton of places, including high network jobs. It would take a decent sized amount of evidence to convince me that all of those other places were wrong and basically only Netflix was right to use BSD in network heavy cases.
My suspicion is that either:
1. The difference is minimal to negligent, and not enough to justify mixed OS development
or
2. The difference is significant, but you have to be pushing your network much harder than most teams ever do for it to show up.
FreeBSD networking during WhatsApp and Netflix starting era was definitely better than Linux. Both in their specific usage pattern. So it wasn’t some ideological reason but actual technical one. But that was 10+ years ago.
If I remember correctly from reading Meta / Facebook successfully switch all of WhatsApp large bare metal 4U server to their standardise Linux blade by 2017. They also spent 3 years working on Erlang BEAM and Networking so it was working close enough.
Netflix continues to use FreeBSD and it is working well for them. Both the lead engineer comment and submit update on HN frequently. Do a search and it should be easy to find it. ( on my phone now so can’t quote links )
2022 now I think any general advantage of FreeBSD over Linux in networking performance would likely be minimal. But it also doesn’t make sense to switch to another for the sake of OS unification.
And say what you will, something like 20% of all internet traffic has a FreeBSD endpoint. And doing 400Gb/s of encrypted streaming from one box is quite an accomplishment.
I would argue the reason someone like Netflix or any of the other large orgs using FreeBSD come there is for the simplicity/cohesiveness. If you're looking to do something like in-kernel TLS or something, way easier on something smaller, documented, and with an OS devel team that will likely incorporate your work in future releases.
It is amazing. I keep following their work from 100Gbps, 200Gbps, 400Gbps and now looks like 700Gb/s [1]...... woah
https://github.com/freebsd-net/netperf/blob/master/Documenta...
What's holding this back is legal issues, since ZFS's licenses is not compatible with the GPLv2.
Don't assume that this means feature parity, not by a long shot. ZFS on FreeBSD affords you native NFSv4 ACLs (hugely important in corporate settings), tight integration with jails, much better delegation support, the FreeBSD boot loader is actually updated to understand all the ZFS features, and offers boot environments and checkpoint rollbacks right from the boot loader menu (and not having to shuffle around with a dedicated /boot partition like you do on Linux).
If none of those features appeal to you, ZFS on Linux will seem very much the same. Indeed, within the limitations of Linux itself, ZFS does very well at providing every feature that Linux lets it provide.
But to imply that these are empirical differences between FreeBSD and Linux, well, that's nonsense.
> Like all other systemd-nspawn features, this is not a security feature and provides protection against accidental destructive operations only
Doesn't seem very similar to jails to me.
I don't know about Facebook-driven efforts specifically offhand, but I recall attending a talk at LinuxCon 2015 _specifically_ about optimizing the network stack. Given by a Red Hat employee, IIRC.
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
From a quick glance through I see a lot of work on eBPF stuff, memory, I/O subsystem, cgroups, and btrfs.
I would not be surprised if there still were some cases where FreeBSD could do better. The network stack in particular has a vast amount of performance heuristics and tuning where you can just happen to hit the right or wrong side of some performance jump depending on your exact case, for example.
But to go from there to the network stack is more performant is puzzling. There are certainly cases where the Linux network stack is more performant than FreeBSD -- search for linux vs freebsd network performance and you'll find benchmark tests and complaints and anecdotes around the place including freebsd forums and mailing lists where FreeBSD is slower, and yet FreeBSD proponents would (rightly) say these are not proof Linux's network stack is faster and there might be all sorts of reasons for the differences. So you can't have it both ways.
Netflix uses FreeBSD for some things yes. Not sure if number of big internet corporations using an OS would be a favorable metric for FreeBSD either. Again you can't have it both ways. Netflix says FreeBSD is capable of performing well in that kind of environment, it does not say that it outperforms Linux in general or even in that environment. Any more than Google says the opposite.
I don't really disagree with any of the stated reasons, but I also didn't see a reason that would make me even consider making the move with our servers, or even bother with some small number of servers. At least for me, I'd need a bunch of REALLY GOOD reasons to consider a move like that. A huge cost savings AND some huge time savings in the future might do it.
Likewise, some people and many businesses see the immediate now and aren't always the best at planning for long term, and/or are overly optimistic that their pain points will eventually be fixed.
But that's the thing. The pain points mentioned in this article aren't really that strong to need to ditch the OS and move to a new one. These kinds of decisions have huge tradeoffs.
One could argue, in fact, that moving to a non-traditional OS will make it much harder to hire experts or hand off the system to another team in the future.
After CentOS Stable was cancelled, I migrated a number of our platforms to FreeBSD because it met our needs and I enjoyed working on it. No surprises, nothing breaking, and most importantly, no drama.
Nothing there seems to deliver explicit customer value when switching to FreeBSD.
How will switching help you deliver your service? Or is it just a "nice to have thing"?
If your backend OS starts to run software more efficiently, cost less to host, has less maintenance downtime, has fewer security incidents, has fewer crashes, etc, having changed to it has produced customer value.
Is the service working, is it not? (are there show stopping OS issues?)
Are the performance characteristics we need there, or not?
Is the service secure or not?
There's a whole lot of things right through that entire space where the choice of operating system can make quite a fundamental difference.
As a general statement, I wouldn't want to make this kind of a migration without something bordering on a killer feature that wasn't possible otherwise, or some fundamental driver problems (I've replaced Linux with FreeBSD on border devices in the past, due to particular issues with Linux in ways I couldn't afford to have keep biting me)
But they left out one of the bigger reasons, IMHO. FreeBSD doesn't tend to churn user (admin) facing interfaces. This saves you time, because you still use ifconfig to configure interfaces, and you still use netstat to look at network statistics, etc; so you don't have to learn a new tool to do the same thing but differently every couple of years. Sure, there's three firewalls, but they're the same three firewalls since forever.
Personally I blame all of the major Unix-derived operating systems (Linux, macOS, BSDs), as none of them show any interest in standardizing any APIs or commands invented this millennium. The subset that’s common to all of them is frozen in time, and is slowly being replaced by new bits that aren’t. From epoll/kqueue to eBPF, containers/jails to seccomp/pledge, DBus/XPC to init systems… from low-level network configuration (ifconfig, ip) to high-level network configuration (whatever that is this month).
On FreeBSD, the last commit on it was last week.
They're not the same tool, and FreeBSD didn't abandon their core tools, because they're part of the base system.
https://sourceforge.net/p/net-tools/code/ci/4030929bb6f3ee6f...
It's still used in some user spaces, and the Linux kernel supports the system calls for net-tools still today, because of the Torvalds Prime Directive, "though shalt not break userspace".
> 70 binaries just for initialising and logging
It’s just not true. Those 70 binaries provide much more functionality than an init system, they can cover a significant portion of system management, including a local DNS resolver, network configuration or system time management. You can dislike the fact everything is so tightly integrated (which feels ironic given that the post goes on to praise a user space from one team), but let’s at least be correct about this.
Do. Not. Want.
So, I have this pile of 70 binaries that are inexplicably tied to my init system, and (in my, informed enough for me, opinion) they're all garbage. How do I remove them without breaking init?
This month's fresh hell: I have a problem where the systemd replacement for xscreensaver (logind, maybe? Good luck finding the culprit, let alone the manual!) won't accept my password unless I exit the current X session with "switch user" then restore the session using the normal login screen.
There's a whole section on JWZ's xscreensaver page (from over a decade ago) explaining how to avoid this class of bug, but what does he know?!?
That reminds me; I wonder if a *BSD is a good enough daily driver for the pine book pro yet (it's probably easier to port their kernel than to fix Linux userspace, after all...)
Depending on distro, all can be substituted for alternatives (assuming they're used at all, I've seen a number of distros package the kitchen sink "just in case")
Nothing about using systemd as pid1 required using any of their other developed tools, even logind. Find a distro that doesn't use them, or roll one yourself.
It's a mistake to see "Systemd" as a single application - it's a collection of developed-together (so tend to work slightly nicer together) userspace tools - like FreeBSD without the libc and kernel, or GNU coreutils or similar. People don't seem to complain too much about those "Bundling Everything Together".
Docker, podman, lxc, lxd, etc are userland components. Linux has cgroups and namespaces.
FreeBSD jails are a bit more complicated because FreeBSD isn't distributed the way Linux is. Linux is distributed as just the kernel, whereas FreeBSD is a base OS. This probably could've been phrased better as, "Linux has no interest in userland and I want some userland consistency". That's fair, Linux was built around the idea that operating system diversity was a good thing long term, FreeBSD was more interested in consistency. I'm reading between the lines, a bit, here because of the critique of SystemD (note: not all linuxes use SystemD)
Personally speaking, I like both Linuxes and FreeBSD but I don't think debating the two is important. Rather, I'd encourage turning your attention to the fact that every other component on a system runs an OS-like interface that we don't make open OS's or "firmware" for.
This is really telling for the level of analysis done: systemd has been the target from a small number of vocal complainers but most working sysadmins only notice it in that they routinely deal with tasks which are now a couple of systemd stanzas instead of having to cobble together some combination of shell scripts and third-party utilities. Confusing noise with numbers is a dangerous mistake here because almost nobody sits around randomly saying “this works well”.
To sum up: healthy FLOSS competition is welcome and needed.
There's too many decades of cruft and backwards compatibility built up. Afaict, most interesting new OS's being built right now are similar to UNIX, but very explicitly not.
In terms of innovations, I'd bet MacOS has evolved too. Although "logical partitions"-like solutions were already known for some time, Linux made it widespread through containers; io_uring allows high throughput syscall-less zero-copy data transfer and futex2 allows to implement NT synchronization semantics that are very common in game development. All that ignoring just how much the desktop changed!
The UNIX children are definitely not sitting still.
Then again they would probably charge 10x as much for a 2U rackmount server where the main difference was a nice stainless steel front fascia...
This may not be the best place, but I have to. I'd like to tell you that, although we've had some disagreements on HN, I carry no bad feelings and, for as much as possible, have extreme respect for you and your opinions.
I'm glad our previous disagreements don't prevent us from posting when we do agree.
So it ok to agree to disagree. :)
Where is this coming from exactly? The linked article about Facebook is 7 years old. The following benchmark shows the exact opposite: Linux's network stack has long surpassed FreeBSD's. And I would expect nothing else given the amount of work that has gone into Linux compared to FreeBSD.
https://matteocroce.medium.com/linux-and-freebsd-networking-...
https://netflixtechblog.com/serving-100-gbps-from-an-open-co...
However, I'm up to serving 709Gb/s of TLS encrypted traffic to from a single host to real Netflix customers with FreeBSD.
Basically just wondering if both of you are correct and the default implementation of FreeBSD is in fact losing to other flavors of Linux but the networking customized OS' (not even just thinking of Netflix at this point) are still superior.
Most of the changes that I have for the 700g number are changes to implement Disk centric NUMA siloing, which I would never upstream at this point because they are a pile of hacks. They are needed in order to change the NUMA node where memory is DMAed, so as to better utilize the xGMI links between AMD CPUs.
Or is that Single CPU? Because I thought you try to avoid the complexity of NUMA. Which I remember it was causing issues with first gen Zen.
Its mostly an experimental platform, never intended for production, to shake out issues that we'll see when next generation platforms show up with DDR5 and PCIe Gen5. With a current single-socket setup we'd be limited by memory bandwidth around 500Gb/s or so, but 800Gb/s should hopefully be within reach from a single socket with DDR5 based servers. Especially since the limiting factor now is the traffic on the xGMI links between the sockets is unbalanced.
I'm glad there are some people who use and prefer FreeBSD and other init system now, because diversity in digital ecosystems is benefits the whole just as diversity in natural ecosystems do.
The shot taking at systemd here was disingenuous though. The author complained about the number of different systemd binaries and the lines of source code, but all these tools provide a highly consistent "system layer" with standardized conventions and high quality documentation-- it's essentially the same argument made to support FreeBSD as a large body of kernel and userspace code that's maintained in harmony.
Nothing wrong with the comparison probably, but I got the impression the author just really wanted to do the migration and found some reasons to do so, without actually needing it. Nothing wrong with that as well. It's just the expectations set by the title that are off
I think many of us (including me) have a tendency to try to quickly generalise our experiences, even when it's not appropriate - and when we go onto explain things to others without the original context it can sound too abstract or come across as evangelical. Either way, it loses meaning without real examples.
If FreeBSD jails and Solaris zones were equivalent to Linux containers, we'd have seen them take over the backend already. We haven't. They're really useful, they provided a degree of safety and peace of mind for multi-tenancy but they're not granular enough for what's done with $CONTAINER_RUNTIME these days.
Jérôme Petazzoni has an old talk where he touches upon container primitives and compared them to jails : https://www.youtube.com/watch?v=sK5i-N34im8
Are there any FreeBSD-centric answers to Ceph or Gluster or Lustre or BeeGFS?
Might be a future alternative in the space.
https://gitweb.dragonflybsd.org/dragonfly.git/blob_plain/HEA... was last updated in 2018, and at the time much of the clustering logic is described as "under development" or "not specced."
It's not BSD centric but it works.
https://docs.freebsd.org/en/books/handbook/disks/#disks-hast
I've been using Linux since I was a teenager, so it's not like I am a stranger to fixing driver issues, but it seemed like no amount of Googling was good enough for me fix these problems (googling is much harder when you don't have functioning wifi). As a result I've always just stuck with Linux (or macOS semi-recently, which I suppose is kind of BSD?).
I've read hundreds of post surrounding this issue. I've used laptops for decades but I don't think I have ever used this feature. It doesn't take even a second to hit a couple keys and sleep a laptop. The only feature lower on my priority list would be syncing the RGB keyboard to soundcloud. But that's just me. Evidently the lid-close-sleep thing is of vital import to millions of laptop users. It's one of those things where I just shake my head in bewilderment.
hw.acpi.lid_switch_state=“S3”
> Evidently the lid-close-sleep thing is of vital import to millions of laptop users.
It's one of many little things that takes a laptop from feeling like some weird barely-functioning misfit of a toy to an actual portable tool that you don't have to worry about or baby.
I set up Hyper-V for the first time last summer, and I seem to recall it just working.
My take: I love FreeBSD as a server os. It's really, really well designed. Spend a bit of time with the handbook and after a while it's really simple to hack on. I really like the separation of the base OS, I like the init, I like jails, documentation puts every Linux distribution to SHAME.
But laptop?... Unless you scour for exactly the parts that will work (esp for wifi), you're going to have a bad time, even on old equipment. At the same time as I rebooted my server, I had an old ThinkPad I decided I wanted to FreeBSD for the hell of it. I gave up on it. It may have been possible but it was just too much work. In this day and age when almost any Linux desktop distro boots without a hitch on hardware that's not completely brand spanking new, it was just not worth it
Usually when this happens in Linux, a few hours of Googling, swearing, and retrying is enough to fix my problems, and I'm sure that with enough time that approach would have worked in FreeBSD as well, but I always grew impatient.
> almost any Linux desktop distro boots without a hitch on hardware that's not completely brand spanking new
Can't speak for anyone else, but even for brand spanking new hardware, as long as I've stuck with AMD drivers, nowadays Linux "Just Works" when I boot it. Obviously YMMV between systems, but I feel like Linux has finally become competitive with Windows and macOS from a usability standpoint.
Honestly I just wish there was a way to run ZFS as a core partition on macOS. APFS is pretty good, but a full on ZFS thing is really cool, and makes FreeBSD so much more appealing as a result.
Maybe I'll just back up to S3 or something. You have peeked my interest.
I've had a hard enough time in the past with out of tree drivers on Linux. I can't imagine what it's like with an OS vendor as hostile to the concept as Apple is.
Some of the distros are very good, but some of us who have work to do cringe at the thought of bringing up newer versions of an OS just to check all the things that've broken and changed needlessly.
There were other choices that Linux could have taken rather than sysvinit vs init.d and some distros through the years have taken different approaches. FreeBSD's rc.d is not sysvinit either.
I don't think that is true. If you look at those other distros you'll mostly find that it either didn't pan out, or it's quickly approaching the same design and architecture of systemd, where distros start using declarative prgramming to integrate tooling around common workflows. This is inevitable when you consider the constraints on the system as a whole. Other related service management tools like Docker are under the same constraints and you'll notice those have a similar architecture too.
for instance, the systems better checks all memory banks and disks before it finishes booting.
Having a system boot slower is an annoynance when downtime occurs, but having a system boot incorrectly is far worse.
I've run a lot of Linux machines on a lot of hardware for a lot of years and have no idea what could have been going on that would cause a 5+ minute boot, that was also unnecessary enough that a different BIOS could get it down to 30s. Back in the bad old days of circa ~2000 when I ran Linux on a variety of garage-sale potatoes, I don't think any took 5 minutes to boot.
Their first reason is really saying a lot but with few words. For one, there’s no systemd. The init system is maintained alongside the entire rest of the system which adds a lot of consistency. The documentation for FreeBSD is also almost always accurate and standard. Etc etc
I think you also largely don’t need a docker or etc in it since jails have been native to the OS for decades. I’d want to do some cross comparison first though before committing to that statement.
Shouldn’t be lost that the licensing is also much friendlier to business uses. There’s afaik no equivalent to rhel, for that matter. This goes both ways though as how would you hire a FreeBSD admin based on their resume without a rhce-like FreeBSD certification program?
Edit-I’ll posit that since FreeBSD is smaller an entity wishing to add features to the OS might face either less backlash or at least enjoy more visibility from the top developers of the OS. Linus, for instance, just has a larger list of entities vying for his attention on issues and commits.
On that playing field, the integrated nature of FreeBSD is nice but it’s an asterisk on top of the kernel rather than anything approaching what makes up a the system part of an Operating System. Almost everything else comes from a third party (and I’m fine with that.)
I haven’t used FreeBSD as a daily OS for over a decade though. What’s the new coolness?
I think the point still stands though. There has been lots of stabilization work done to BTRFS, and anyone using production-recommended features should consider the filesystem stable.
I slap Linux on my servers and desktops and I deploy FreeBSD via pfSense on firewalls.
Sometimes I do experiments and try out BSD on the desktop which hasn't worked out yet for me but I live in hope because I adore *BSD as much as I do Linux.
If BSD is the way to get your servers to do what you want then lovely. Do it and remember you have choice.
When I compare to ubuntu which is the OS I use for linux mostly now: * kvm is superior to bhyve in every way * automating security updates via apt are better than a combination of freebsdupdate/pkg updates. Plus the deb packages are made by Ubuntu and just work. ports/pkgs are third party on freebsd * rebootless kernel updates exist for ubuntu * It is easier to find people familiar with linux right away
Really though the learning curve of freebsd <-> linux is not high.
Attempts were made to fix it, but they eventually decided to ship 13.0 without wiregaurd.
It was very policitcal because the company that sponsored the development had already promised the feature to customers.
Even if an OS is 'better', a VM with a short lifetime will generally be 'good enough' very quickly. If you add a very large ecosystem and lots of support (both open source and community as well as commercial support) and existing knowledge, FreeBSD doesn't immediately come to mind as a great option.
If I were to go for an 'appliance' style system, that's where I would likely consider FreeBSD at some point, especially with ZFS snapshots and (for me) the reliably and fast BTX loader. Pumping out BSD images isn't hard (great distro tools!) and complete system updates (due to the mentioned "one team does the whole release") are a breeze as well. This is of course something we can do with systemd and things like debootstrap too, but from a OS-image-as-deployable perspective this will do just fine.
If your disk is slow or dieing, you might blame UFS, but it's not really UFS.
I've had some vague issues with the I/O scheduler, which isn't really UFS, but at the same time, UFS may be the only real client of the I/O scheduler, I think ZFS does it's own thing, anyway the systems were UFS only. This is super vague, and I don't have more details, but I just want to put it out there. For one class of machines that had a lot of disks (about 12 ssds), did a pretty even mix of reads and writes, evenly spread across the disks, upgrading from FreeBSD X to X + 1 wasn't possible because there was a large performance reversion. I think this was 10 -> 11, but it's possible it was 11 -> 12. Because this came up while my work was in progress migrating to our acquirer's datacenter which included switching to their inhouse Linux distro, it made sense to just leave those hosts on the older OS, and not spend the time debugging this. We didn't have a way to test this without production load, but that had user impact, and it would take a while to show up. It's quite possible this was just a simple tuning error, or possibly a bug that has been fixed for some time; the symptoms were obvious though: processes waiting on io, but the disks had a lot of idle time.
If you have a lot of files in a given directory, that's kind of slow, and IIRC, if the directory every had a lot of files, the extra space won't get reclaimed until the directory is deleted, even if most of the files are unlinked. (This isn't uncommon for filesystems, some filesystems handle it better than others, there are application level strategies to deal with hashing/deeper directory trees)
If the filesystem ever gets too full, the strategy to search for free space changes to one that's less fast; it won't change back, so don't fill your disk too much. (This ends up being a good idea for SSD disk health too, and again isn't super unusual in filesystems, but some filesystems probably do better). tunefs(8) says:
> The file system's ability to avoid fragmentation will be reduced when the total free space, including the reserve, drops below 15%. As free space approaches zero, throughput can degrade by up to a factor of three over the performance obtained at a 10% threshold.
UFS has snapshots, which is great, but everyonce in a while, you end up with a snapshot you forgot about, and it can really eat disk space and you may miss it. Not really a performance issue, but can lead to overfilling your drive.
Of course, there's the obvious that UFS has no support for checksumming, but that's not performamce. Soft updates do allow for some amount of consistency in meta data, and background fsck is nice (but could tank performance, I suppose).
for instance, what is your ARC configuration in this case? It can have a massive impact of performance.
getting ZFS to perform well takes a bit of work, but in my opinion performance is on par with most filesystems. (and it has a ton of additional features).
L2ARC can improves cached reads, but it's not magical, especially not for random reads... or writes. (and yes, I know about SLOG, but doing async is faster than improving sync)
And don't get me started on how ZFS is not using mirrors to improve read speed (unlike mdadm can do, cf the difference between o3 n3 f3) or how it can't take advantage of mixed arrays (ex: a fast NVME + a regular SSD or HDD to add redundancy: all the reads should go to the NVME! The writes should go async to the slow media!)
If you don't have a RAID of fast NVMe that are each given all the lanes they need, you may not see a difference.
But if you are running baremetal close to 100% of what your hardware allows, and the choice of everything you want to buy and deploy, you'll see these limits very soon.
In the end, I still chose ZFS most of the time, but there are some usecases where I think XFS over mdadm is still the best choices.
I haven't made much use of them but the mirrors or raidzs seemed to perform more or less inline with expectations (consumer hardware may not have the PCIE lanes really available to run multiple fast NVME devices well).
Compare with XFS over a mdadm say in raid10 3 legs f3, then cry.
> consumer hardware may not have the PCIE lanes really available to run multiple fast NVME devices well
Trust me, I have all the lanes I need, even if I would always wish I had more :)
direct i/o?
This latest version has something called "journaled soft updates" and it's a metadata-journaled system, i.e. the actual data is passed through, and it's non-CoW.
> I sometimes experienced severe system slowdowns due to high I/O, even if the data to be processed was not read/write dependent. On FreeBSD this does not happen, and if something is blocking, it blocks THAT operation, not the rest of the system.
I’ve seen this for a long time in Windows, where any prolonged I/O brings the entire system down to its knees. But it also seems to affect macOS (which is based on FreeBSD) as a system, though it’s not as bad as on Windows. Has Windows improved on this over the years? I’m unable to tell.
Curious to know the reason.
Reason I want to know is when I tried running ubuntu off a spinning disk and wanted to move some games to the drive it would transfer about 1gb then lock up. Didn’t happen with an SSD.
Computers get faster, we throw more at them.
Physics is still the law of the land.
Truh truh truh trade offfss; in computer eng; truh truh truh trade offs sorry Bowie
That's somewhat overstated, so I wouldn't draw such conclusions. (https://wiki.freebsd.org/Myths )
That being said it also forced me to use FreeBSD a lot more than I ever would have otherwise and I have a lot of respect for the OS itself. I would not use it everywhere but it has amazing latency which makes it obviously great for networking.
Related: https://www.youtube.com/watch?v=o_AIw9bGogo -- The tragedy of systemd.
Where Benno Rice (FreeBSD Committer / FreeBSD Core member) explains the value of something like systemd.
"FreeBSD's network stack is (still) superior to Linux's - and, often, so is its performance."
This is wrong, if it was the case most large compagnies would use BSD, atm they all use Linux, the only large compagny using BSD is Netflix because they added some tls offloading in kernel for their CDN which could have been done in Linux btw.
imo don't use tech that is not widely used, you're going to reinvent the wheele in a worse way because tool a.b.c is missing.
so kinda windows with wsl v2 is the way to go
https://www.cl.cam.ac.uk/teaching/1516/ConcDisSys/2015-Concu...
I run FRR Routing punching 400gbit+ @ 75% CPU load on FreeBSD 12.
On the same hardware, Debian fell over at 1.8gbit.
I have been using FreeBSD for networking for awhile now but not at those levels
My question is how well moving their systems will shake out longer term.
You should know that due to license incompatibilities (CDDL and GPLv2), it's not really as smooth as the BSD integration (I wonder how Canonical avoids this issue). You know that OpenZFS's codebase is a monorepo (for the most part) but the in-kernel implementation is vastly different.
I use Zfs as my boot file system on FreeBSD and Arch Linux and both work great. Linux was a bit of a faff to set up though, FreeBSD was easy.
There's a significant difference between almost all devices and essential devices like storage: you need to have the storage online as soon as possible, graphics are purely optional in the boot process. If you only use ZFS as "external" storage, sure it's effectively not different but as a boot drive?
I guess to each their own but I dislike docker. I think it’s bloated and over complicated.
There's nothing wrong with switching to another OS, but implying it's because the other OS is somehow empirically "better" is misguided.