Hacker News Reader

Comment by acatton | Hacker News Reader

Funny enough, I decided to play with FreeBSD for personal projects in 2020. I gave up and I am reverting all my servers to Linux in 2022, for the opposite of the reasons mentioned in this article.

* Lack of systemd. Managing services through shell scripts is outdated to me. It feels very hacky, there is no way to specify dependencies, and auto-restarts in case of crashes. Many FreeBSD devs praise launchd, well... systemd is a clone of launchd.

* FreeBSD jail are sub-optimal compared to systemd-nspawn. There are tons of tools to create freebsd jails (manually, ezjail, bastillebsd, etc…) half of them are deprecated. At the end all of your jails end up on the same loopback interface, making it hard to firewall. I couldn't find a way to have one network interface per jail. With Linux, debootstrap + machinectl and you're good to go.

* Lack of security modules (such as SELinux) -- Edit: I should have written "Lack of good security module"

* nftables is way easier to grasp than pf, and as fast as pf, and has atomic reloads.

View on HN

I've grown to love systemd. It solves a lot of my problems with init scripts, particularly the ones involving environment / PATH at boot time. I've made init scripts for things before which work when they are invoked manually, but not at boot time because PATH was different. With systemd I am confident that if it works through systemctl it will work at boot.

Maybe I am not tuning Linux appropriately, but I have been in situations where a Linux system is overwhelmed / overloaded and I am unable to ssh to it. I have never had that experience with FreeBSD -- somehow ssh is always quick and responsive even if the OS is out of memory.

Most of the systems that I deal with are Linux, but I still have a few FreeBSD systems around and they are extraordinarily stable.

I think there's something pathological in the I/O subsystems on Linux that make it a bad experience - I've experienced horrible U/I latencies on desktop and server settings with Linux when there was any kind of I/O load, and found FreeBSD to always be a breath of fresh and quick air in this regard.

There are definitely pathological cases around. Here's an 8 year old bug, still valid, that's a common extreme slowdown when copying to/from a USB drive:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1208993

It sometimes manifests as audio stuttering (the kind which sounds like unstable sample rate) which suggests a lot of context switching, perhaps a scheduler problem. Had this problem intermittently and seemingly at random for many years. Sometimes would happen all day, then nothing for months. Quite Bizarre.

For a long time after creating my own Linux distro, I had the same kind of problems also. It turns out the Linux kernel is horribly tuned by default. After a number of tweaks and adjustments, I finally got all those bugs ironed out. Now my (four core) desktop is perfectly smooth and responsive under all loads, even playing video and running multiple builds while copying files around. Here's the important parts of what I've done:

* set disk i/o schedulers to 'bfd' for spinning drives and 'deadline' for solid state, and 'none' for nvme, by creating a file in /etc/udev/rules.d . kernel must have deadline and bfd schedulers compiled in.

* turned on SCSI block multiqueue in kernel config. requires kernel command line option scsi_mod.use_blk_mq=1 to actually enable it. this helped, but did not completely cure the disk i/o problem.

* patched kernel source file ./block/blk-mq-sched.c to hard limit number of queued block device requests to 2, instead of default which is like 32. this absolutely cured the problem. no more disk i/o dragging the system down. doesn't seem to have a major effect on throughput.

* kernel is configured for full preemption, with 1000hz timer frequency.

* for architectures which will boot using the muqss cpu scheduler patch, i enable that with a 100hz timer freq instead.

* overcommit is disabled, as well as swap, and i use earlyoom to ensure process destruction proceeds in a controlled manner in event of memory exhaustion.

That's the bulk of it. No real magic involved; just un-fuck-ifying the default kernel config, which is garbage even for server use IMO.

(This is on a 4.x kernel btw, and I have no plans to downgrade to the 5.x series.)

The fact that these some or all of these tweaks aren't done by default would seem, along with other evidence, to support my belief that Linux is actively being sabotaged by people who don't want it to succeed.

Software have a hard time keeping up with hardware architecture, mostly because of backwards compatibility...

Imagine running a restaurant, normally you can take 32 orders and have the customer sit and wait. One day you get new chefs that can make food 100x faster, but now you can only take one or two orders before the chefs have the food ready and you have to give it to the customer that ordered it. So despite the chefs being 100x faster it now takes much longer to place an order, and the waiting line can grow long with impatient customers.

> I've grown to love systemd.

I think I’ve grown to appreciate it, but not love it. It feels like both systems are at opposite ends of a pendulum.

The RC/unit system was very approachable for me. You could explain it easily, and you could get inside of it and mess around very easily. That affordance/discoverability was awesome.

With systemd, simple things are nicely templated. I can find an example and tweak it to achieve what I want. Complicated things get complicated real fast.

The RC system was simple, I'll give you that.

I love not having to track down pid files. I love not having to check if the pid file contents match a valid instance of the expected binary.

The RC system was simple and also full of exceptions and corner cases.

What is the use case for that? I never had these needs on servers. Maybe you are doing something very different from me, so I am curious. I use RC on servers and systemd on clients; things run robustly for 15+ years on many servers for me (with security updates included).

Let's say that you're starting Apache Tomcat. You have to dig into the tomcat startup scripts and figure out where, if anywhere, it writes the pid file to disk so that you can later use it to determine if the process is running or not. If java happened to crash, though, this pid file is stale and might not point to a valid process. There's a chance that the pid has been reused, and it could have been reused by anything -- even another java process!

This is important, because this pid file is used to determine which process receives the kill signal. If you get it wrong, and have the right permissions, you can accidentally kill something that you did not intend to kill.

This is further complicated if you want to run multiple instances of Tomcat because now you need to have a unique path for this pid file per tomcat instance.

If the thing that you're trying to run doesn't fork, you then have to execute it in the background and then store the result of $! somewhere so that you know how to kill the process later on.

It's all very error prone and the process for each daemon is often different.

> Let's say that you're starting Apache Tomcat. You have to dig into the tomcat startup scripts and figure out where, if anywhere, it writes the pid file to disk so that you can later use it to determine if the process is running or not.

https://tomcat.apache.org/tomcat-8.5-doc/windows-service-how...

The commandline argument is --PidFile and --LogPath. Most, if not all, programs allow you to customise this. It should never be a guessing game, especially when you are the one creating the init file, therefore you are the one in control of the running program.

Those arguments are for the Windows service and don't appear to have a corresponding Linux option. On the Linux side, if things are still done the same way as they were in the past, you had to set CATALINA_PID before starting the java process.

It's still a guessing game, though, even with CATALINA_PID. It is entirely possible for Java to crash (something which RC scripts do not handle, at all) and another java process starting up which happens to be assigned the same process id as the dead java process. This can not happen with systemd units because each service unit is its own Linux cgroup and it can tell which processes belong to the service.

You could just run every daemon in its own jail. You don't need to chroot (unless you want to), and don't need to do any jail specific network config (unless you want to), but you could use the jail name to ensure no more than one tomcat (or tomcat-config) process, and you could use the jail name/id to capture process tree to kill it without ambiguity.

With respect to servers that crash, approaches differ, but you could use an off the shelf, single purpose daemon respawner, or you could change the whole system, or you could endeavor to not have crashing servers, such that if it crashes, it's worth a human taking a look.

Sure, you can do all that and set it up manually. I'd love to be corrected here but last I checked this was not done automatically in any BSD. Systemd recognizes this is so common that it does it automatically for every service using the Linux equivalent (cgroups). IMO now that we have these tools, every sysadmin is always going to want to use cgroups/jails for every service all the time and never want to use pidfiles because pidfiles are tremendously broken, error-prone and racy.

Even with systemd one may need to deal with pid files for services with more than one process.

Systemd has heuristics to detect the main process to send the kill signal or detect a crash, but it can guess wrong. Telling it the pid file location makes things reliable.

Plus creation of a pid file serves as a synchronization point that systemd uses to know that the process is ready. The latter is useful when the service does not support systemd native API for state transition notifications.

And the best thing is that even if systemd has to be told about pid files, one never needs to deal with stalled files or pid reuse. Systemd automatically removes them on process shutdown.

A pid file is never actually reliable though. Since the supervised process has control over it, it can write any number it wants in there and trick the service manager into signaling the wrong process. As long as the process is not root and can't escape its own cgroup, systemd's pid detection is going to be less error-prone in basically every case.

I can't stress this enough. Pid files are really bad. The fact that we used to have to use them is a flaw in the OS design. Using them for startup notification is also a hack in and of itself. The kernel has enough information to fully track lifecycle of these processes without writing information into files that are inherently prone to race conditions, and we now have better APIs to expose this information, so we shouldn't need to use these hacks anymore. I don't think there is any Unix-like system left that considers the old forking daemon style to be a good idea, and systemd's handling of pidfiles is really just a legacy compatibility thing.

How about Apache, Nginx, pulseaudio or ldapd?

Your description is fine, but it’s missing one crucial detail: it’s specific to Linux. In FreeBSD this is already taken care of by rc infrastructure; there is no need for the user or sysadmin to mess with it.

I fully agree that it is a Linux only solution, but that's kind of the topic here.

This still doesn't handle restarting crashed services, and it still is true if you need to make init scripts for your own services outside of the ports tree.

It took me a long time to come to terms with systemd, but I am very glad that I did. For me it makes defining services both easy and reliable.

Not the solution - the problem is specific to Linux. In FreeBSD pidfiles are something taken care of a decade ago, you don’t need to think about them.

We're almost in agreement -- FreeBSD does handle pidfiles better and more uniformly than Linux does. It doesn't help you if you want to run two instances of the same service.

I looked at one of my FreeBSD servers to see how it was handled in the case of ZNC, an IRC bouncer that I use. ZNC doesn't produce a pid file on startup, so the FreeBSD RC framework tries to find a matching process in the output of 'ps'. [1] As soon as you attempt to run multiple instances of a given service, this falls over completely. [1]

daemon(8) helps -- it handles restarting of processes if they crash, and it can manage pid files and prevent them from getting stale. Nothing on my FreeBSD system uses it. The unbound (dns cache) port uses the pidfile option for rc.subr(8). Looking at how check_pidfile is implemented, it attempts to verify that the process represented by the pidfile matches the process name. Pids also wrap on FreeBSD, so you have a chance of a false positive match if you run multiple instances of a given daemon. I could, of course, change unbound's rc scripts to use daemon, but that feels like a lot of thinking about pidfiles for something that was taken care of a decade ago.

I do like FreeBSD, don't get me wrong, and I use it in my every day life. systemd solves problems for me, and I really like the way it manages process groups by utilizing kernel features.

  1. https://docs.freebsd.org/en/articles/rc-scripting/#rcng-daemon
  2. https://unix.stackexchange.com/questions/503150/rc-scripts-for-multiple-zope-instances-in-freebsd

You have to do it for correctness. Every use case needs this handled.

Okay so you have a PID file somewhere /var/myservice/myservice.pid. The contents of that file is a number which is supposed to correspond to a process you can find in proc.

But PIDs are recycled or more likely your PID files didn't get cleaned up on reboot. So you look at your file and it says 2567, you look up 2567 and see a running process! Done right? Well it just so happenes that a random other process was assigned that PID and your service isn't actually running.

pidfd's are the real real solution to this but the short and long tail of software uses pidfiles.

How about keeping pidfiles on tmpfs mounts so they do get cleaned up? I guess that'd be an organisational thing to get all the apps to change to a consistent /var/tmp so you could mount it..

That's still not good enough. pids wrap and eventually it could point to something valid, especially on a busy system.

systemd uses linux cgroups so that it knows exactly which pids belong to the group.

The defaults have surely changed over the years, but pid_max used to be ~32K by default. On the system I'm typing this comment on, /proc/sys/kernel/pid_max is set to 4194304.

> The defaults have surely changed over the years, but pid_max used to be ~32K by default. On the system I'm typing this comment on, /proc/sys/kernel/pid_max is set to 4194304.

The commit which changed the defaults is this one: https://github.com/systemd/systemd/commit/45497f4d3b21230756...

How does systemd make sure it never reuses a cgroup, OOI? If you're worried about PIDs wrapping, surely that applies to anything randomly-generated as well.

It's derived from the service name, which is guaranteed to be unique. It's similar to how names need to be unique in /etc/init.d/

Wait... how would pid wrapping impact anything at all unless the process died and the pid file was stale (which is usually a fairly unusual circumstance - I've had it happen maybe a dozen times over the last couple of decades). And I guess if you have stuff surprise dying without any of the usual cleanup, you have a pretty serious situation in general. Maybe need a process monitor service just for that :)

I was focusing on the "surprise startup" scenario where it would all get wiped and reset.

As with anything else, your chances increase with the number of processes you have running.

If it blindly reads the pid and sends a kill signal, your odds are pretty good with a limit of 32K pids on a busy system. If it confirms that the process name matches an expected value, you have less of a chance.. but if your process name is bash, java, or python, maybe not as good as you would hope.

I don't have things crashing a lot, but it's naive to pretend that it never happens. It could result in two things: the rc system telling me that everything is fine or the rc system sending signals to some unrelated, poor, unsuspecting, processes.

I don't need a process monitor just for that. I have systemd. :-)

> But PIDs are recycled or more likely your PID files didn't get cleaned up on reboot. So you look at your file and it says 2567, you look up 2567 and see a running process! Done right? Well it just so happenes that a random other process was assigned that PID and your service isn't actually running.

If you're unlucky, though, pid 2567 might match another myservice instance. This can easily happen if you're running many instances of the same service. Even checking /proc/$PID/exe could give you a false positive.

I don't expect many programs do this (and I agree the real solution would be handles) but it should be possible to check the timestamp on the PID file and only kill the corresponding process if its startup time was earlier.

There might still be race conditions but this should cut down the chance dramatically.

Your nice shell-scripts in /etc/rc.d just won't handle a service crashing for any reason, at all (systemd does that)

Your nice shell-scripts in /etc/rc.d will start EVERYTHING and ANYTHING just in, case, even if you don't always need it (systemd does support socket activation)

Your nice shell-scripts can't handle parallelism (systemd can)

Your nice shell-scripts can't reorder stuff at boot, you have to specify it by hand (systemd can via Needs/RequiredBy etc)

Your nice shell-scripts are worthless if you need to run more than one instance of a service per server (with systemd having many instances of the same service is a feature, not an after-thought)

Your nice shell-scripts won't help you troubleshoot a failing service (systemd will, via systemctl status AND journalctl).

SysV-RC handles all that apart from socket activation, which is the job of inetd. This is all before systemd.

I must say that now it has more features and the configuration format, while could be simpler, is way better than sysvrc magic comments. I'd suggest it should be XML with a simple config file equivalent for simple cases, as the subtree update in YAML-like is a hell anyway.

Service restart has been a solved problem in UNIX for over 40 years. It is called inetd. It is not perfect, and it was originally created for a slightly different operating model, but it does restart crashed «services». They are, in fact, called daemons. Shell scripts are for manual service starts, stops and restarts (when the daemon does not support a restart on SIGHUP), they are not meant for the automated service restarts.

Different UNIX systems have gone on to replace inetd to address specific shortcomings of the inetd model with launchd, service management facility, systemd, SAM, but the service restart is not an innovation that systemd has brought along. It is a further, fine-grained, improvement over a decade old concept and a tool + ecosystem that has implemented the concept (inetd + rc scripts).

If your service crashes, either your service or your build sucks.

OpenBSD does this stuff right. You keep pushing faulty and buggy crap over and over.

This speaks more to the issues regarding the "made of many little pieces without coordination" spelled out in the linked post than to using simple scripts for service management. Also the idea that you would have pid files scattered all over instead of /var/run seems like the sort of chaos linux spawns.

The *BSD rc systems are robust and rely on a framework that abstracts stuff out, you're never going to be digging in there for a pid file's location.

I looked into how the RC system works on one of my FreeBSD servers. You might be surprised to see how rc.subr does this -- it parses the output of 'ps' to look for the process name and matches it to what's listed in the script. This only works in the simplest of cases. If the script lists a pidfile, it finds that pid and makes sure that the process name of that pid matches what is in the script.

You don't need to dig for the pid file's location, I'll give you that, but it's also possible for this robust rc subsystem to kill the wrong process under the right circumstances.

My system runs unbound (dns cache), so I looked for its pidfile. It's not in /var/run, but rather wherever it happens to be specified in the config. The default is /usr/local/etc/unbound/unbound.pid.

In mainstream Linux distributions, things in /var/run are likely sane if you live in the packaged world. My tomcat example was about adding software outside of the packaged world / ports tree, which is why you would have to track down the pid file to make a service. Given that unbound places its pid file outside of /var/run, which is more chaotic?

I like FreeBSD, and I use it daily! I also very much like the features and organization that systemd brings to the table.

  1. https://cgit.freebsd.org/ports/tree/dns/unbound/files/unbound.in?id=6eb036d0d6656a72d27b34557ed4bf1feb1cd4f0

> The RC system was simple, I'll give you that.

A lot of the staying power of Unix is that it's based on a very small set of simple, yet powerful, concepts. RC is one of them.

Complicated? https://lwn.net/Articles/701549/ If you have time, read the first two articles mentioned early on.

All the problems with NFS dependencies described in the article existed before. The system administrator was supposed to address them manually by ordering things in rc files adding sometimes various sleep pauses to wait for things to start.

Systemd allowed to solve them in a reliable way at the distribution level with no need for the system administrator to do anything beyond providing the exports and fstab entries.

I think RC was only simple if you stuck to a single distro and had a neatly predefined pattern of work.

It's true that systemd is nowhere near as easy to hack/get into but at the same time I find that the largely consistent definition language/CLI give me a lot less reason to want to do so.

When the need does arise, there's usually a line or two I can add to my service definition to get it to do what I want.

I couldn't say the same about init scripts.

On my personal linux workstation I experienced multiple times that I couldn't ssh into it after the desktop froze, it was due to too small swap file.

That's interesting, what proportion are you using to determine swap size? For example 1x RAM, 2x RAM, etc.

I run at 1xRAM. Seems that my problems went away after that.

Ubuntu by default sets it to 2GB even if you have something like 16GB RAM. Swap file can then fill up pretty quickly.

> Ubuntu by default

Yep, one of my systems got this treatment. It's annoying because OOM has been an issue ever since. I'm playing with EarlyOOM and trying to remember if it'll be a huge pain to resize the partition for more swap space. Thanks for your reply.

Thanks for the tip on EarlyOOM. Swap file is at least easier to handle than swap partition.

Some run their Linux without swap at all, however I think that is better for a server setup where you run a small set of specific set of binaries, thus you know your load.

That is not the case on a desktop where you running a variable set of binaries depending on your task.

I’m somewhat surprised how bad this works on Linux, I don’t think I ever experienced a swap problem on Windows (not even in the 9x days), I guess this is because "Year of the Linux desktop" never happened, Linux is primarily a server and embedded OS, not a desktop OS.

On my desktop I run with no swap, overcommit disabled, and earlyoom. I frequently do heavy work with it including such things as Chromium builds and it's trouble free. See above for the tweaks I've made to ensure perfect responsiveness under all loads.

Unless one have the latest SSDs on a server to tolerate occasional spikes in memory zram makes more sense. Just configure it to use lz4 compressor. This way even with halve of the memory compressed the system remains somewhat responsive and if the compression no longer helps, then killing the memory hog is probably the right thing.

1x RAM plus a bit on any machine I want to hibernate eg a laptop.

Anything else, 2GB or less or not bother with a swap disc and just use a swap file with a cap at say 2x RAM. Also depends on how much RAM and disc is available.

Have you tried to enable zram or zswap?

If it makes you feel any better, I have been using Linux for like 25-30 years now and I still experience "sigh. and now I can't SSH into the box to kill the thing I shouldn't have started".

My boss graduated from Berkely, and so I occasionally had to administer a FreeBSD box he kept around for the "superior OS philosophy".

My biggest annoyance, apart from the obvious lack of systemd, was a social one: Any time I had to look up how to accomplish some non-trivial task I would inevitably find a thread on the FreeBSD forums by someone else who has had exactly the same problem in the past, together with a response by some dev along the lines of "why would anyone ever need to do that?".

Even though I severely dislike systemd, and am a fan of FreeBSD's stability and simplicity, I also have found this attitude to be an annoyance. For example, if I want to upgrade a fleet of a hundred or so FreeBSD boxes remotely, the answer seems to be, "Don't do that. You must upgrade each one by hand."

This is the principal reason why I've leaned toward Debian stable (which also aims at being a stable UNIX like FreeBSD) for decades, even though Debian has also been infected with systemd and has made other questionable decisions. Alternatively, I've also had good luck with Void, Alpine, and Artix. (I've had difficulties with electron apps on Void desktop, but it runs flawlessly on servers.)

> For example, if I want to upgrade a fleet of a hundred or so FreeBSD boxes remotely, the answer seems to be, "Don't do that. You must upgrade each one by hand."

either that or "don't do an upgrade, build new ones and replace the old ones instead"

> (I've had difficulties with electron apps on Void desktop, but it runs flawlessly on servers.)

If by any chance it was related to sound, today is your lucky day as I just solved that this morning before work.

If you're getting the "do it by hand" answer, you're asking in the wrong places.

Ansible, SaltStack, all the devops automation stuff is available and can certainly handle "freebsd-update fetch; freebsd-update install; reboot".

I needed to script something for a FreeBSD server, and I was doing the development in Linux. It would have been OK if I had used Ruby/Python/Perl, etc., but I decided to do it in Bash. I had to make quite a few fixes when I got the script on the FreeBSD box. Then I deployed it to an OpenBSD system. More fixes.

Did you try "pkg install bash"?

> I would inevitably find a thread on the FreeBSD forums

That was your mistake. With BSDs, you use the man pages ;-)

I have mixed feelings about this. On one hand, this is extremely alienating. On the other hand, I respect when devs reject features to keep their code simple.

I don't think this has anything to do with graduating from berkely, I'm a hobo with a keyboard.

> At the end all of your jails end up on the same loopback interface, making it hard to firewall. I couldn't find a way to have one network interface per jail.

You may want to look at vnet, which gives jails their own networking stack; then you can give interfaces to the jail. If you use ipfw instead of pf, jail id/name is a matchable attribute on firewall rules; although it's not perfect, IIRC I couldn't get incoming SYNs to match by jail id, but you can match the rest of the packets for the connection. And that brings up the three firewalls of FreeBSD debate; maybe you had already picked pf because it met a need you couldn't (easily) meet with ipfw; you can run both simultaneously, but I wouldn't recommend it. Nobody seems to run ipf, though.

Edit: you may also just want to limit each jail to a specific IP address, and then it's easy to firewall.

And iocage makes all of this extremely simple, it can manage IPs for you and all that nonsense. There's also a sort of "clone" of iocage for bhyve.

> Many FreeBSD devs praise launchd, well... systemd is a clone of launchd.

No they are not. Systemd has a way bigger scope then launchd. It's like saying a truck is the same thing as a family car because they both solve the mobility problem.

> FreeBSD jail are sub-optimal compared to systemd-nspawn.

systemd-nspawn isn't a container. For example it doesn't manage resources such as CPU or IO. Again, the scope is way different and in this case there is a whole slew of things systemd-nspawn isn't going to manage for you.

BTW launchd doesn't have a feature like 'systemd-nspawn'.

> Lack of security modules (such as SELinux) -- Edit: I should have written "Lack of good security module"

And how is SELinux a good security module? SELinux with it's design fell in the same pitfall dozens of security systems did before it; ACL-Hell, Role-hell and now with SELinux we also have Label-hell.

> systemd-nspawn isn't a container. For example it doesn't manage resources such as CPU or IO. Again, the scope is way different and in this case there is a whole slew of things systemd-nspawn isn't going to manage for you.

It does[1]. At the end systemd-nspawn runs in a unit, which can be (like all systemd unit) resource-controlled.

> BTW launchd doesn't have a feature like 'systemd-nspawn'.

Neither does systemd. systemd-nspawn is just another binary, like git ant git-annex. The only difference from git and git-annex is that systemd and systemd-nspawn are maintained by the same team.

But like git and git-annex, most systemd installations don't have systemd-nspawn, but systemd-nspawn needs systemd.

> And how is SELinux a good security module? SELinux with it's design fell in the same pitfall dozens of security systems did before it; ACL-Hell, Role-hell and now with SELinux we also have Label-hell.

SELinux is generic enough to allow for sandboxing of services. But in its mainstream use, SELinux is well design that it allows for reuse of policies. Most people don't care about SELinux, they just run CentOS, install RPM from the repository, and everything works out of the box with heighten security. (= if there is a vulnerability in any of these packages, the radius-blast of the attack is limited thanks to SELinux's Mandatory-Access-Control)

[1] https://www.freedesktop.org/software/systemd/man/systemd.res...

> but systemd-nspawn needs systemd.

That's not, strictly speaking, true. systemd-nspawn is able to do the cgroups-based containers fine without systemd as init, though you do lack some of the fancier resource control and networking features.

> Systemd has a way bigger scope then launchd.

I hear it's taken over dns. Has it taken over sound too?

When will it be able to read mail?

systemd the init system has nothing to do with DNS.

systemd the family of tools has a DNS server, systemd-resolved. Like all other tools in the family, using the init system does not require using the other tools, and sometimes also vice versa.

In the broader "Poetteringware" family of tools, sound is handled by pulseaudio, though the server side of pulseaudio is in the process of being replaced by pipewire.

I'm sure someone is about to bring up logind.

Is it pulseaudio or systemd-pulseaudio now? :)

> systemd the init system has nothing to do with DNS.

And btw is that why systemd using distros lock up for minutes on boot if the machine doesn't have internet connectivity?

> systemd-nspawn isn't a container. For example it doesn't manage resources such as CPU or IO.

False.

> And how is SELinux a good security module?

It's good enough for most environments requiring RBAC. Additionally, it's complemented by the excellent sandboxing ability of SystemD unit files.

>Lack of systemd. Managing services through shell scripts is outdated to me.

This, this and this again !

After decades of cron, I discovered systemd timers via a passing comment I read here on HN.

My god is it amazing. No more hacky work-arounds in my scripts, systemd now takes care of all the magic such as random timings etc.

I'll never go back to cron.

Can you elaborate? I'm a macOS laptop/FreeBSD server guy. What are systemd timers, how do they work, and why do you feel they solve your problem better?

Generally I would say the biggest difference is that with timers, you get more control.

For example, you can schedule a timer to run 10 minutes after boot. Or a timer that actives 10 minutes after it has last finished running (note: not when it started last time but when it finished! So if the proc takes 10 hours, there is a 10 minute gap between runs. If it takes 10 minutes, there is still a 10 minute gap).

You can also schedule something 10 minutes after a user logs in (or 10 seconds later, etc.).

Additionally you get Accuracy and RandomizedDelay. The former lets you configure how accurate the timer needs to be, down to 1 sec or up to a day. So your unit now runs somewhere on the day it's supposed to run. And with the later you can ensure that there is no predictable runtime, this can be important for monitoring.

My biggest favorite is Persistent=. If true, systemd will check when the service last ran. And if it should have been scheduled atleast once since then, it'll activate. I use this for backing up my home PC. When I do a quick restart, no backups are done but when I shutdown for the night, first thing in the morning my PC has a backup done.

Counterpoint is that I've never had a requirement for timing scenarios like this, so now I'm dragging all that along for the ride for no advantage. Bloat is great when you need that rare thing I guess; otherwise it's just bloat and complexity for no benefit.

Maybe because you've never had the ability to use such timing requirements. Having more advanced tools available also makes you think more in them. If you only ever used cron, there has been no reason to think about a service running every hour or immediately if it missed the schedule. Because the only tools are every hour and on reboot. And no option to make "every hour" mean "between script invocations" instead of "on the hour of the schedule".

This was my experience. Cron was great because it did what I needed it to do and I knew how to use it. But over the years I accumulated all sorts of hacks to avoid pitfalls. When I learned systemd timers I didn't like the complexity, but as new needs arose I thought about both cron and systemd and realized that systmd timers were better for 75% of my needs.

There are lots of things that I have the ability to do that I've never had the requirement to do. I call those things "bloat" because they are unnecessary features that add complexity and bugs to the system.

Unnecessary to you maybe but not to other people. It's a tiny bit rude to simply call a feature rude just because you don't use it. After all, by that argument I could call the screen reader software or the Speech-to-Text bloat. But they aren't.

All of those systemd features have been implemented hundreds of times as buggy hacky shell scripts launched by cron.

agreed, and more importantly it's possible to implement these scenarios with cron if they're needed. i'll take a simple building block over bloat. where does the bloat stop? i can come up with tens more scenarios that timers doesn't cover.

I had hard requirements for randomizing cronjobs plenty of times. Once you have a large number of systems it's the default rather than the exception.

In FreeBSD see the "-J" and "-j" options to cron.

Does that add a static random delay on bootup or does it randomly delay the job each time? Because for timers you can pick that for every single timer individually.

anacron has @roboot, I use it in my cron.d/

It will also run things that were scheduled to run when the server was turned off.

Yes but anacron is still a bit less powerful than timers, since I can schedule a timer to run every hour instead of once an hour plus on reboot in two very short config lines. And I don't think anacron has a concept of running things every hours not including runtime, only including runtime.

you just drop your script in /etc/cron.hourly.

That runs it once an hour not every hour, there is an important distinction there.

Please elaborate on why this might matter? I can't think of any scenarios.

It helps dispersing jobs like backups. If you have 20 servers, your backup storage is not hit by them all at the same time. Even better, because it takes into account how long it runs, if two servers hit it at the same time, the reduced performance will automatically spread the server that took longer behind the other. That scheme reduced the needed capacity in network and CPU of our backup servers a lot.

If it ran every hour, all servers would either hit the backup store at the same time or you would have to manually disperse them. Randomized Delay is great if you want to avoid this problem with short running jobs but it doesn't work well when most jobs take 10 minutes or more and the delay becomes larger than the timer's repeat interval.

So in that case, using a "run every 60 minutes" scheme is a massive advantage that reduces coordination needs.

Then you'd put that in cron.d/ something like 5 * * * * would run at 12:05, 1:05, 2:05, 3:05, etc...

And every server will have to be put on a different schedule. By running it every 60 minutes, it doesn't matter when it runs, it automatically aligns. Once you managed 200 servers, that becomes valuable and no longer manually managable ("How many servers are running backups on :05? Do I need to move some? Can I fit in one more?")

Not OP, but this page seems to have a nice writeup: https://wiki.archlinux.org/title/Systemd/Timers

Fcron can also do random timings, jitter, running if the time passed occured while system was off, timing based on runtime not elapsed time, delay until system is less loaded, avoid overlapping instances of the same job.

http://fcron.free.fr/doc/en/fcrontab.5.html

1.0 released 21 years ago.

What about anacron?

Good point. I'm not familiar with it.

I agree with this, don't have much experience, but I figured systemd timers out, very fast.

> Lack of systemd. Managing services through shell scripts is outdated to me. It feels very hacky, there is no way to specify dependencies, and auto-restarts in case of crashes. Many FreeBSD devs praise launchd, well... systemd is a clone of launchd.

I'm not a fan of systemd personally but I do understand it has some good parts to it (such as the ones you've listed). That all said, you can still specify dependencies in FreeBSD with the existing init daemon. Albeit it's a little hacky compared with systemd (comments at the top of the script). But it does work.

> FreeBSD jail are sub-optimal compared to systemd-nspawn. There are tons of tools to create freebsd jails (manually, ezjail, bastillebsd, etc…) half of them are deprecated. At the end all of your jails end up on the same loopback interface, making it hard to firewall. I couldn't find a way to have one network interface per jail. With Linux, debootstrap + machinectl and you're good to go.

It's definitely possible to have one interface per jail, I've done exactly that. However back when I last did it (5+ years ago?) you needed to manually edit the networking config and jail config to do it. There might be a more "linuxy" way to do this with Jails now though but its definitely possible. eg https://etherealwake.com/2021/08/freebsd-jail-networking/

There's a lot to unpack here. For example, there's certainly other ways to network jails and all three ways you've mentioned to maintain jails are not deprecated.

Security modules do exist, they're different from Linux. Are you sure you're not just expecting FreeBSD-as-Linux?

As for init... What can I say, I've never been anti-systemd, not even remotely, but rc.d is much nicer than sysvinit, and I find it much simpler to understand than systemd. In fact, I think rc.d is an example of how Linux could have alternatively migrated from sysvinit without pissing some people off.

> Security modules do exist, they're different from Linux. Are you sure you're not just expecting FreeBSD-as-Linux?

You're right. My original message was wrong, I edited it, while keeping the original content. What I meant is "good security module".

SELinux on CentOS is "enabled by default and forget about it", unless you do something weird. MAC (= Mandatory Access Control) on FreeBSD requires much more configuration. They have some cool stuff like memory limits, but it's not as powerful as SELinux.

The security posture is quite different, so it's not as easy as just oh, turn on some magic module and be done. Security requires work, regardless of OS.

A fair bit is included in the base system already, see capsicum for example. Also, see HardenedBSD, which is arguably better than anything Linux has built-in.

>which is arguably better than anything Linux has built-in.

No it isn't. There's a reason why government and military servers run hardened Linux with SELinux, and not any of the BSDs.

Any source for this statement?

There are no DoD STIGs[0] for the BSDs, meaning they cannot run on military servers. Similarly, there are no CIS guides[1].

What would I know, only designed and implemented TS+ systems for a decade!

[0] https://public.cyber.mil/stigs/downloads/

[1] https://www.cisecurity.org/

There are a lot of them for Windows... should I then trust that linux is as secure for government use as Microsoft systems are?

Now you're shifting the goal posts after I provided sources that the US gov does not use BSD.

Microsoft Server systems for government use have been audited and have strict controls for implementation, hardening, securing, etc.

They're probably more secure than BSD.

Yet I see BSD on the STIG list as well. Are there different levels of security for the list items or how does this work if Windows can be said to be more secure than BSD even if both are checkmarked on the audits?

Since you keep editing your comment:

Not sure what you mean, but the custom Windows Server that the US government uses is likely more secure than BSD.

>Yet I see BSD on the STIG list as well.

No you don't. DISA does not provide STIGs for any of the BSDs. The US government does not use the BSDs for secure systems (TS+ etc.).

Interesting... does that mean that all secure systems are running on Cisco networking?

As said, they do have STIGs and CIS documents for JunOS but I guess they don't run any Juniper in the US secure networking despite of having certifications.

>Interesting... does that mean that all secure systems are running on Cisco networking?

Now you're strawmaning on top of shifting goal posts again.

>As said, they do have STIGs and CIS documents for JunOS

No they don't. The CIS documents specifically outline the operating systems they support, and there are ZERO BSDs listed (click on Operating Systems)[0].

Similarly, there are zero STIGs for Junos OS. There are STIGs and CIS benchmarks for Juniper network devices, but not for Junos OS. The actual devices could be running on FreeDOS for all we care, but FreeDOS in and of itself would not be allowed to run on any servers. Hilariously, even Juniper is moving away from BSD. Junos OS Evolved is Linux based.

You're fractally wrong.

[0] https://www.cisecurity.org/cis-benchmarks/

It’s not about providing any real security, it’s about ticking checkboxes. So yes, if the checklist says “Linux and Windows are ok” then you can mark the checkbox, and with FreeBSD you couldn’t.

I said built-in, you seem to have missed that part. SELinux is not built-in(though it is for certain distributions of Linux).

Security is hard to define, let alone prove. Everyone has a very different definition of security. So first one has to ask, secure from what?

I imagine most of the reason around BSD not on the official list(s) is because it's not as popular. I mean GenodeOS[0] is arguably one of the most secure OS's around these days, but I doubt you can find any public Govt support(by any govt) for running it in production today.

Going back to my original comment, security is complicated, and there is no "secure", but hopefully for a given set of security threats, there is a "secure enough".

The same exists in physical security. Our home door locks are notoriously not secure, but they are generally secure enough for most home needs. But your average home door lock would obviously be idiotic as protection for Fort Knox's gold deposit door.

Comparing BSD to Linux security is complicated, but for most high value targets, the answer probably is, run more than one OS. Root DNS servers and other highly critical internet infrastructure all do this as a matter of common practice. If you are mono-culture Linux only, I worry for your security, as you are effectively a single zero-day away from being owned. Linux, BSD, Windows, etc will all have RCE's and zero-days as a normal part of existing.

0: formal proof secure(sel4), for some definitions of provable even: https://genode.org/

>I said built-in, you seem to have missed that part.

I did not miss that part, you're just mistaken.

>SELinux is not built-in(though it is for certain distributions of Linux).

Wrong. SELinux is 100% "built-in" to Linux. That's like saying btrfs or Wireguard are not "built-in" to Linux because certain distros may or may not have them compiled in. Nonetheless, SELinux is part of the kernel [0].

The rest of your dribble is a painful Gish gallop because you were decisively proven wrong. Mature up a bit and take the L. Fomenting about being proven wrong is against the Guidelines here.

[0] https://lore.kernel.org/selinux/

I'm not trying to hate on SELinux, it's great stuff, for what it is. I'm not trying to hate on you either, though clearly you seem to have hatred towards me, which is just sad.

I'm happy to accept that SELinux is now built-in to Linux, the kernel parts do indeed seem to be built in now, news to me, thanks for that. I don't follow Linux kernel stuff much anymore, I haven't contributed to Linux in over a decade.

You seem to assume SELinux is the end-all be all of Linux security. It isn't. I recognize, based on your other comment, that you are fairly new to the field(a whole decade, go you!). Please open your mind and accept differing perspectives, it will do wonders for your ability to reason about security properly.

HardenedBSD[0] essentially implements grsecurity for FreeBSD, plus FreeBSD has built-in capabilities with Capsicum[1], which is true capability based security, which is much different than SELinux's MAC stuff. If you don't believe me, go read the capsicum paper[1] and come to your own conclusions, it might prove enlightening.

Also, see CheriBSD. :)

0: https://hardenedbsd.org/content/easy-feature-comparison 1: https://papers.freebsd.org/2010/rwatson-capsicum/

If you just want to continue hating on me, no reason to respond, we can go our separate ways. If you want to have a reasoned discussion about security, then I'm happy to continue.

>You seem to assume SELinux is the end-all be all of Linux security.

Never said or implied anything of the sort.

>I recognize, based on your other comment, that you are fairly new to the field(a whole decade, go you!).

I've been implementing secure, hardened UNIX and Linux probably longer than you've been alive. I just specifically worked on DoD TS+ systems for a decade.

The rest of your Gish gallop is nonsense. Linux also has capability based security ON TOP of all the other aspects of security, SELinux included.

>If you want to have a reasoned discussion about security

That's not possible with you. You instantly showed how little you know about security in general when you flouted your lack of SELinux knowledge, then you proceeded to Gish gallop and sealion because you've been called out.

Stop it.

If this is how you interact with people in the real world, I feel sorry for them. Grow up and interact with humanity better.

>If this is how you interact with people in the real world, I feel sorry for them.

IMAX-levle projection. The only one here you should feel sorry for is yourself.

> Grow up and interact with humanity better.

"no u". Grow up and take the L with grace.

Maybe he's just tired of people who offer ignorant opinions and argue based on conjecture and not actual knowledge.

DING DING DING DING DING

And they say BSD users have attitude issues, lol.

I also use BSD, AIX, Solaris, etc. As someone stated below, maybe I'm "just tired of people who offer ignorant opinions and argue based on conjecture and not actual knowledge."

>>SELinux is not built-in(though it is for certain distributions of Linux). > >Wrong. SELinux is 100% "built-in" to Linux. That's like saying btrfs or ???>Wireguard are not "built-in" to Linux because certain distros may or may not >have them compiled in. Nonetheless, SELinux is part of the kernel [0].

It really depends on what you mean by "SELinux". The core kernel bits of SELinux, are of course, part of kernel by definition. However, SELinux is not really useful unless it comes with the SELinux policy definition which defines what applications do "normally". This work needs to be done by the Linux distribution, because without it, it's much like relationship between software and hardware. "Without the software, it's just a paperweight."

>It really depends on what you mean by "SELinux".

Not it doesn't. Just like it doesn't "really depend" on what you mean by "btrfs".

> However, SELinux is not really useful unless it comes with the SELinux policy definition which defines what applications do "normally".

"However, btrts is not really useful unless it comes with the userland utilities to actually make and manipulate file systems"

See how pedantic this is? Not only are you being pedantic, you're still wrong, fractally so.

That is exactly operating system does, which is the topic of discussion. Linux OSs such as RHEL as an example.

> This work needs to be done by the Linux distribution

Which are defined as operating systems by CIS and STIG.

This is a poor attempt at "DO YOU MEAN GNU/LINUX?".

Stop it.

You can boot a system with a btrfs root file system without having the btrfs-progs installed.

Good luck trying to use SELinux without having the policy installed; it's guaranteed to be non-functional. And given that the SELinux policy is distro-speicific, it's not like you can take a random Linux distribution, and enable SELinux and expect it to work. You enable SELinux on the boot command-line, but without the policy installed, it will be dead in the water. And configuring the SELinux policy is extremely non-trivial. It's several orders of magnitude more challenging than running, say, "mkfs.btrfs".

>That is exactly operating system does, which is the topic of discussion. Linux >OSs such as RHEL as an example.

If that's your definition of an OS, then there are plenty of Linux distributions --- aka, an "OS" by your definition --- that do *NOT* have SELinux built in, because they don't have an SELinux policy defined that will work with that distribution's system daemons.

Therefore, by your definition SELinux is not "built in" to all versions of Linux (specifically, "distributions"). Q.E.D.

>You can boot a system with a btrfs root file system without having the btrfs-progs installed.

Wrong again! You can actually have Linux systems that do not support booting from btrfs, but have btrfs-progs installed. Or systems that have neither.

>Good luck trying to use SELinux without having the policy installed

They are installed and included in the Linux operating systems used by the US government, as I stated above. This is a non-point.

> And given that the SELinux policy is distro-speicific, it's not like you can take a random Linux distribution, and enable SELinux and expect it to work.

Wow, it's almost like it be nice if there were standards used by the government and other organizations looking to secure their operating systems. Maybe they can form an agency, I'll call it the Defense Information Systems Agency. They can make standards that secure and lockdown systems, and make sure SELinux is configured properly... We'll call these Security Technical Implementation Guides, STIGs for shor... Oh wait...

> And configuring the SELinux policy is extremely non-trivial. It's several orders of magnitude more challenging than running, say, "mkfs.btrfs".

Immaterial to the matter at hand.

>If that's your definition of an OS

A distribution is an operating system by definition. It's not my definition, it's the definition[0].

>that do NOT have SELinux built in, because they don't have an SELinux policy defined that will work with that distribution's system daemons.

Still "built-in" to Linux. Whether or not it's enabled or complied in is an implementation detail, but it's still "built-in" to Linux operating systems, most definitely those used by the government for secure systems, which was the original point. Thanks for proving my point, Q.E.D.

>Therefore, by your definition SELinux is not "built in" to all versions of Linux

Using your illogic, there are BSDs that send your password through plaintext over the wire because they only have rlogin. They don't have SSH "built-in".

SELinux is absolutely "built-in" to the Linux kernel and operating systems. Whether or not specific implementations of Linux have it compiled and enabled is besides the fact that it is built-in, not third-party.

You're fractally wrong again. Take the L and stop while you're this far behind.

[0] https://en.m.wikipedia.org/wiki/Linux_distribution

I agree with your point, but disagree with the way you said it. Also, it's drivel*, not dribble.

“Government and military servers” tend to run Windows ;-) SELinux looks nice on paper - another box to check - but it’s just another mitigation later, not something that can be considered “trusted”.

Microsoft Server systems for government use have been audited and have strict controls for implementation, hardening, securing, etc.

They're probably more secure than BSD.

>SELinux looks nice on paper - another box to check - but it’s just another mitigation later, not something that can be considered “trusted”.

This is fractally wrong.

> there is no way to specify dependencies

    # PROVIDE: mumbled oldmumble 
    # REQUIRE: DAEMON cleanvar frotz 
    # BEFORE:  LOGIN 
    # KEYWORD: nojail shutdown

* https://docs.freebsd.org/en/articles/rc-scripting/

* https://www.freebsd.org/cgi/man.cgi?query=rcorder

From your link https://www.freebsd.org/cgi/man.cgi?query=rcorder

> The `REQUIRE' keyword is misleading: It does not describe which daemons have to be running before a script will be started.

> It describes which scripts must be placed before it in the dependency ordering. For example, if your script has a `REQUIRE' on `sshd', it means the script must be placed after the `sshd' script in the dependency ordering, not necessarily that it requires sshd to be started or enabled.

“Managing services through shell scripts is outdated.” By inference, since most things in Unix like systems are built on the notion of shell (and automation using the shell), this is saying large part of the foundation of Unix is outdated. A tool is a tool, but I would take a shell script from rc.d any time over a binary blob from systemd.

systemd unit files are text files, not binary blobs. And it is much easier to grok a unit file than a 500 line init script.

At what cost? Thats not the enirety of systemds complexity curve.

Your systemd unit file is backed by pages and pages of docs that must be comprehended to understand and hack on. Unix developers have all they need from the script. Furthermore, its all in context of existing unix concepts and thus your unix experiance is paying dividends.

Now you're just exaggerating. Are you really saying spending 5-10 mins skimming through a couple of man pages is that hard? Are you saying that a lot of documentation is a bad thing? (I thought FreeBSD fans liked to harp about their handbook..) And besides, there are already hundreds of systemd unit files on your system that you can easily copy and make relevant changes for your own services. Not having to deal with finicky shell features is a major advantage IMO.

I think the disconnect we're having is your inability to perceive complexity. And I don't blame you, it's not easy to quantify. I suggest you start with, Out of the Tar Pit by Ben Moseley. I'm not knocking on documentation, it's a vital property that I consider when adopting new technology.

What I'm saying is that systemds documentation currency (if you accept my metaphor) is spent on covering its accidental complexity and it's voluminous. If you disagree with me, that's fine. This is just my experience as a linux user that's had to deal with systemd.

If your claim is that systemd man pages are well written documentation then I think you're exaggerating and I'll wager you've relied on stackoverflow examples or tutorial blogs to solve your systemd issues--because I have. The reason for this is because the number of concepts and abstractions that you have to piece together to solve your problem is massive. But yeah, it's just a 5 line Unit file. I prefer strawberry kool-aid, thanks.

I genuinely don't see what's so complex about a service unit file. It's a simple INI file that has multiple sections that describes the service, tells what command to run and specifies any dependencies. It's literally the same thing that init scripts do except in a much more concise and efficient manner. And as I said before, there's a ton of systemd service unit files on any Linux system that you can take a look at and use as inspiration for your own services. Taking a little time to learn the ways of systemd is not a huge burdensome task like you're making it seem to be. I don't see why you think everyone should conflate systemd with complexity.

And about the voluminous documentation, well man pages are supposed to be comprehensive and cover every single aspect of the tools being described. They're not there to just be an intro to systemd for new users and administrators. If you want something like that, look no further than the "systemd for Administrators" series of articles written by the systemd author himself. https://github.com/shibumi/systemd-for-administrators/blob/m....

> I genuinely don't see what's so complex about a service unit file

It't not the unit file that's the problem, it's the mountains of junk, low quality C code written by an obnoxious, arrogant twit named "Linux Puttering" who has proven for 15+ years he couldn't care less about code quality or system reliability.

Besides the anecdotes shared by others over the years about the horrible experiences they've had with systemd, I have one of my own to share. When developing my own distro to escape the bloated, laggy hell that is Ubuntu, I started the build on my existing Ubuntu system. I found out the hard way that accidentally double mounting virtual filesystems on the target volume causes systemd to crash the system after about 60 seconds, with no possible way to recover. On MY system, with no junky ass systemd, making this error harms nothing at all and can be easily fixed.

The people who talk about "buggy, hacky" shell scripts appear to be some of the same type of people who shrink in horror from the idea of compiling their own kernel, or working at the command line. (i.e. not really "hackers" at all.) There is nothing at all wrong with using shell scripts for startup. It is in fact the simplest, and IMO most elegant way of doing the job, and no it isn't buggy or hacky in the least. The file system is the database and unit file and the already existing shell is the interpreter.

My system starts much more quickly than Ubuntu and is much faster and more responsive in daily use also, so the "startup time" excuse is a myth, and practically all of the other contrived examples people use to justify the use of systemd can be done BETTER using shell scripts in conjunction with small, light weight, single purpose utilities built the UNIX WAY.

It's "just an INI file" but you would have to understand what the thing that's interpreting does. All the stuff that the OP described as a positive - dependencies, auto-restarts, socket activation - somewhere there's a codebase that's implementing all that, and you can't just understand your "config file", you have to understand what that codebase is actually doing with all of its concepts. Elsewhere in this thread someone writes about how great it is that systemd is using a cgroup namespace to keep track of each process instead of a PID, and maybe that is great, but it's yet another new concept that you have to understand to understand how any of this works. Etc.

You could say that a shell script is a config file for bash, and you have to understand bash to understand what it's doing. But a shell is both simpler than systemd, and something that anyone working with Linux already understands.

The equivalent comparison with init scripts would be all the documentation and complexity of every program invoked by the init scripts, not just by sysvinit or rc's documentation and complexity directly. systemd just has most of that built in. And if you're using socket units, the order of what order to start things is essentially outsourced to the kernel, so that's a bit of a simplification.

Try building and maintaining a linux distro without systemd, especially for a large organization that needs to write their own init scripts. And especially when a large number of the devs in that org aren't shell experts, or don't understand the difference between /bin/sh and /bin/bash. And so on.

Here's another example: https://lwn.net/Articles/701549/ before systemd, for complex NFS setups, the sysadmin _had_ to write the init scripts per-site or per-machine. With the solution in the article (systemd generators) one set of unit files shipped by the distro solves the problem for over 99% of users, including most of the aforementioned complex setups.

How dare a system have "checks notes" have too much documentation describing how it works.

FreeBSD has a lot of documentation, which is something people like about it.

I think it actually shows a problem, which is that BSD is designed for all your machines to be special snowflakes with individual names, edited config files, etc instead of being mass managed declaratively. So you need to know how to do everything because you’re the one doing it.

Our research company hosts 15 PB of data on what we call Single System Imaged FreeBSD with ZFS. All systems pull the complete OS from one managed rsync repo during production hours. Doing this for ten years, never ever any problems. Config files are included using the hostname to differentiate between servers. Adding servers doesn't add manual labor, it's the borg type of setup which handles it all.

This is nonsense. Please educate yourself a bit on systems automation in general if you think this is the case.

Is there an automation for "before updating ports, you need to read every single entry in UPDATING in case one of them has a command you need to run after"?

Why is there a chapter on custom kernels under "common tasks" that assumes you're going to have a C compiler and kernel source on your machine and want to installkernel on that same machine?

The unsaid thing here is Linux is largely not used by sysadmin/unix types. Devops has driven this bloat so that people new to the field can just not have to learn any fundamentals about the OS they're building their tools on. For rapid "move fast and break things" VC nonsense, this is a great match. For efficiency, correctness, and long-term maintainability and security, it's a nightmare.

But they only provide fixed functionality, while shell scripts allow for practically unlimited customization.

As for 500 lines - take a look at proper rc scripts, eg the ones in FreeBSD. They are mostly declarative; it’s nothing like Linux’ sysv scripts, which were in some ways already obsolete when first introduced (runlevels? In ‘90s, seriously?)

If you need extra customization capabilities, just run a shell script via the ExecStart= parameter and boom, you have all the power of systemd and the shell combined.

You can even do one better since systemd can natively run rc scripts. If you're on a systemd based distro peak at /etc/init.d. You can even manage services with /etc/init.d and the service command.

The amount of effort systemd went through to make existing software work is genuinely heroic.

Yeah, this conversation seems a bit like people arguing past each other. But it's a result of the fact that the story on Linux was stuck for so long (e.g., sysvinit on Debian, Upstart with some sharp edged hacks on Ubuntu). Systemd as the solution seems to have sucked out all the air out of the room: either it's great and people are idiots, or it's the worst thing on the planet and people using it are sheep.

Yes. Exactly.

But they only provide fixed functionality, while shell scripts allow for practically unlimited customization.

This is the exact opposite of a good thing.

> But they only provide fixed functionality, while shell scripts allow for practically unlimited customization.

Why is unlimited customization a good thing in the context of a system init?

For the same reason it’s a good thing in other contexts. It’s the main reason Unix got popular - because it can be made to fit whatever requirement you have.

If your BSD init script is 500 lines long, you've done something horribly wrong.

That is a flawed argument as the is a big difference between using the shells at places where it makes sense and managing services using shell scripts.

I mean the shell is still used everywhere, like e.g. to configure and control systemd.

Still I would say a lot of core shell tools are indeed outdated (for backwards compatibility).

Large parts of the foundation of Unix are absolutely, obviously outdated, starting from filesystems. There is nothing better yet for big chunks of Unix, but systemd (despite all its flaws) is a notable exception.

Can you expand on "starting from filesystems"?

The fact that filesystems casually allow TOCTTOU races—and that those races cause security vulnerabilities unless you delve into arcane, OS-specific syscalls like renameat2—is an embarrassment.

Username checks out. :D

If I understand correctly, the issue is mutex on file objects at the kernel level. Basically it is a failing of the implementation of the "file" abstraction. Or perhaps a failing of the "file" abstraction itself?

Per Wikipedia [0] (because I had no idea what a TOCTTOU race condition was)

    In the context of file system TOCTOU race conditions, the fundamental challenge is ensuring that the file system cannot be changed between two system calls. In 2004, an impossibility result was published, showing that there was no portable, deterministic technique for avoiding TOCTOU race conditions.[9]

    Since this impossibility result, libraries for tracking file descriptors and ensuring correctness have been proposed by researchers.

It seems to me that solutions to the problem from inside the "files as an abstraction" space won't solve the problem.

I was curious to see how a different abstraction would avoid this problem. Plan 9 FS appears to have had this issue at one point [1], but notably not because of the FS implementation itself. Rather the problem was caused by the underlying system's executing outside of the P9FS ordered access to a given file (someone please tell me if my understanding is incorrect).

Here's an article that talks about the problems of this (and other) race conditions from the level of abstraction [2].

NOTE: I'm not claiming that P9FS is immune from this sort of attack, I'm only commenting on what I've found.

[0] https://en.wikipedia.org/wiki/Time-of-check_to_time-of-use

[1] https://bugs.launchpad.net/qemu/+bug/1911666

[2] https://gavinhoward.com/2020/02/computing-is-broken-and-how-...

Exactly! I think the right way to deal with this is some sort of optimistic transaction support, such as SQLite's BEGIN CONCURRENT or git push --force-with-lease.

> well... systemd is a clone of launchd

sort of. Systemd took a lot of lessons from launchd but also from sysv and upstart. For anyone who hasn't read Lennart Poettering's "Rethinking PID 1" post[1] I highly recommend it. You'll understand the history, but most importantly you'll understand systemd a ton better.

[1]: http://0pointer.de/blog/projects/systemd.html

>At the end all of your jails end up on the same loopback interface, making it hard to firewall. I couldn't find a way to have one network interface per jail.

FreeBSD jails has quite a bit of history (developed in 1998) and made public in the year 2000. FreeBSD has had VNET for use with jails allowing you to add epair interfaces connected to a bridge, or add a physical interface or VLAN to a jail. This feature has been in FreeBSD since 8.0 (2009) and enabled by default since 2018. It also allows you to run PF in each jail.

Sadly many people a bitten by outdated forum and blog posts when it comes to jails.

I agree that firewalling jails with loopbacks is a pain, but most people don't do it that way anymore.

Also PF has always had atomic reloads.

Yeah, it's a weird argument. I've used vnet, but rarely need it. I bind my jails to their own IPs and that's that. That has been available from the very early days of jails.

Again, mature, simple and secure, with very few surprises lurking. There's likely fewer lines of code to support all jail functionality than in systemd.

Regarding jails - I do use separate loopback device per jail, plus pf with nat. No issues with firewalling.

Same .. I don’t use the separate loop back. Just firewall what I need to firewall .

> At the end all of your jails end up on the same loopback interface, making it hard to firewall.

I suppose you didn't use vnet? It's a vastly better jail networking experience. You can pretend jails are separate machines, connected via ethernet. I don't think anyone who knows about vnet chooses not to use it!

> I couldn't find a way to have one network interface per jail.

I think vnet is what you want.

> nftables is way easier to grasp than pf, and as fast as pf, and has atomic reloads.

pfctl allows you to do atomic reloads. `pfctl -f`

Systemd has only one advantage, which is also it's prime disadvantage: It's paws are everywhere in your system. Before there was Systemd, init systems worked okay too - in that space nothing was changed by it.

Every company I worked at (before systemd was mainstream) was running most of their services under supervisord[1] which was started by initv.

I'm not sure initv "worked okay".

[1] http://supervisord.org/

init scripts were "great" if you were a unix wizard. For mere mortals they were frustrating.

1.5M lines of code is not an init system, it is a time bomb... and with dodgy wiring.

I think it was a ZFS dev who complained he had to update 150 files to port ZFS to systemd. Simples?

And you have to love all them binary log files, with their dynamic and spontaneus api. Yes, everything is always new and exciting with systemD.

I use devuan. A debian fork with a choice of init systems (well, everything but systemD. ;) It took the dev/devs 2 years just to swap out/revert the init system.

Also, just learned devuan's website is JS and cookie free. https://www.devuan.org/

Edit: I will concede that systemD is great for a lot of people. I honestly wish them success. The work that goes in to building, maintaining, and using it, is substantial. It must be a boon for the economy and job creation.

Don't make a mistake, systemd is not (just) an init system. It's a replacement of more and more Linux userland. I suppose later on it will replace much of the filesystem, network, and process security management tools, stuff like xattrs, ip / ipfw, and selinux.

I suppose that the end goal of the systemd project is an ability to deploy a production Linux system with just systemd and busybox, and run all software from containers.

Not that it's a bad thing to strive for. But it's not going to be Unix as we know it.

> But it's not going to be Unix as we know it.

Right, that's Plan9.

Plan9 is different in many core concepts. I didn't notice that systemd creates new file systems or allows to mount resources over the network.

(Yes, I do regret that Linux was a hyper-succesful project that replicated Unix, not Plan9. But Plan9 is more complex, maybe it was just infeasible.)

> It's a replacement of more and more Linux userland.

To be more specific is it a replacement of more and more Linux userland that nobody asked. I would be ok if systemd would be a process management system that replaces init with a standardised way of managing services (it would be amazing if it was written a safe language, if it was respecting configuration files, if it did not take over managing system limits, etc. etc.).

Unfortunately it is trying to do too much.

Re nobody asked: I suppose that Red Hat and IBM sales departments sort of did, maybe the embedded teams did, too. Also, the Gnome team forced it on the desktop.

(I'm still able to run Void Linux as my primary desktop without systemd.)

The result of this is that many of the big companies have their own linux distro.

> 1.5M lines of code is not an init system, it is a time bomb... and with dodgy wiring.

https://news.ycombinator.com/item?id=21935186

Oh that's rich. That is rich. The [Flagged] tag is what really makes it. I'm tempted to screenshot that post and make an NFT from it. It really does capture something about the zeitgeist.

Seriously, though, read the article. The 1.2M lines includes removed lines from refactoring. So, it is not completely made up, as implied in that comment. https://www.phoronix.com/scan.php?page=news_item&px=systemd-...

Is this better?

https://twitter.com/pid_eins/status/1214268577509003266

https://www.phoronix.com/misc/systemd-eoy2019/files.html

Including the number of lines in static tables and documentation hardly seems like a meaningful comparison.

I like how userland and the kernel are sync in freebsd. I disdain how linux works in that. However Linux has its core advantages, some distos are well suited and tested for certain scenarios. I used to have a bunch of freebsd, but nowadays, I use containers, light weight runtimes. With Lambdas and Serverless, most, but not all, business api are streamed line where servers matter not. Its the runtime.

Serverless is killing containers. Its killing the need to care about if its freebsd or linux, does it run my api fast enough?

This whole "userland and kernel" thing sounds good, but in my BSD use, the trade-off is that most desktop app packages (and ports) get a whole lot less attention and are more buggy than those in Linux. I imagine it's not a problem for servers.

Btw, HardenedBSD is a good security module and available.

Why exactly is launchd so praised by BSD folks?

Reply on news.ycombinator.com