Systemd 248
raw.githubusercontent.com
raw.githubusercontent.com
I'm not a sysadmin, so I never really felt the hate against systemd. For my use cases it was always not exactly transparent; it was essentially invisible.
Solving problems in multiple different ways is the only reason we have multiple distros serving different needs. Unification is not always a positive thing.
I switched from Windows to Linux 20 years ago to have more choice.
I think it has more to do with the linux kernel being much more advanced than the others, and some of the userspace depending more and more on these advanced features. But I’m not sure about this. Aren’t you referring instead to debian itself providing packages for all of them and the package manager running on each distro? Because I can’t really believe you could just swap out the kernel and be on your way in most cases. If it is the latter, than it was more due to the fact that debian maintainers voted against supporting other systems, because it is a great amount of additional work.
It's a bit more direct than that - previous init systems were kernel-agnostic whereas systemd is explicitly tightly coupled to Linux and udev, and encourages the rest of userland to be tightly coupled to it.
> I think it has more to do with the linux kernel being much more advanced than the others, and some of the userspace depending more and more on these advanced features.
I wouldn't say "more advanced" - in many ways FreeBSD jails are a better technical solution than Linux's various container systems.
E.g. in Gentoo you can choose to use SystemD or OpenRC, but I think that most other distros don't provide such a choice.
And of course after a certain size it has to do with network effects, but it was chosen based on merits — for example debian voted multiple times on the issue and I would say their voting system puts most countries’ to shame.
I believe up until recently the vote on debian was basically use systemd but if the maintainer wants to, they can also support alternative init systems, but there was not much interest. But of course anyone can help the fringe distros on another init systems to make them less fringe.
Yes, I absolutely agree regarding the effort. (not sure about that vote - I honestly didn't follow that discussion)
In parallel, some software (cannot recall - maybe the Gnome window mgr) does rely (in its original form) on SystemD being there => I don't think that's fair, kind of forcing the distros that want to offer it to adopt it => I don't like it - I do see the advantage of having that kind of integration, but I still don't like it :P
The dependency no longer exists because some people took logind from systemd, made it a standalone package and called it elogind lol, and now you can use that instead of logind.
1) eudev works
2) elogind works
3) you can still use sysvinit or literally the multitude of others
The fact that this is so rare it has to be pointed out like this kind of speaks for itself.
> 1) eudev works 2) elogind works 3) you can still use sysvinit or literally the multitude of others.
I am aware of these and use them on nearly all my machines.
Note that both elogind and eudev required a significant amount of work by the Gentoo community when they were pulled out of the systemd codebase.
You may not mind the decisions others make on your behalf, or care about the options that are no longer possible, but that doesn't mean that Linux hasn't lost some of its identity.
Yes, in the sense, that if you want, you can eventually make it the way you want.
No in the sense, that others won't do it for you; they will make linux distribution the way they consider it best, not how you consider it the best.
So if you want to pick and choose the components the way you want and not the way the distributions consider the best, be ready to put in the manhours.
Just like you can mostly expect filesystem behavior to be consistent no matter which distro, so should be the OOM behavior if a "good" solution is found.
The same thing goes for out of memory handling: no size fits all. The kind of handling strategy you need depends on the kind of system you build.
Exactly this kind of flexibility and lack of unification is what made Linux so ubiquitous.
Except with the monoculture of systemd how can experiments be done with other approaches? Nowadays it seems like "use systemd or GTFO".
> Just like you can mostly expect filesystem behavior to be consistent […]
And how many file systems does the Linux support? One can try an approach and see how it works and others can perhaps copy it.
How does another approach towards OOM (or anything else in the kitchen sink that systemd handles) get traction when the 800lb/400kg of RH/IBM is bankrolling The One True Way™.
Is that much different from let's say databases, where "One True Way" was SQL and despite that NoSQL became a thing and is appropriate for some use-cases?
> And how many file systems does the Linux support?
And yet ext4 appears to be the most widespread though. It's true that there are other options such as XFS, ZFS and BTRFS but in general, ext4 is a safe default that works well enough for most use cases unless you have specific reasons to choose otherwise.
Is it that far-fetched to believe that systemd might be the same, and it works "well enough" to be the default and even be bankrolled by some big names?
> How does another approach towards OOM [...] get traction
Presumably the systemd OOM killer can be disabled and something else put in its place if you so desire, just like you can still build or use a distro without systemd or selectively disabling its "bad" parts such as the journal, etc?
Also, just look at the before-systemd landscape, there was much more difference in the repositories, and nowadays even more niche distros can trivially package almost everything since .service files are unified. Also, they are quite flexible, so I don’t feel that anything would be lost.
True, but desktop usage is only one of many things linux is used for. All the "unification" efforts ala systemd try to handle all of those use cases. I don't see why the init system on my server has to be the same as the one on my ultrabook, because only a fraction of what they should do is shared. Having one tool that covers all the use cases just sounds like a very good way to increase the attack surface.
> I believe there are only a handful of people who knows enough to create a proper init system (similarly, sound and graphics) and do it for the community.
I don't think Linux graphics or sound situations have much in common with the init system situation, except perhaps the fact that Poettering tried to fix sound as well.
Well, not even the kernel should be shared —- but most of us is okay with having the same kernel, but at most configured differently. I see it the same way in the case of systemd, of course your ultra book doesn’t need I don’t know how many services, but the basics are very much the same. And to be honest, I’m happy to use the same knowledge/tools on a server machine and a laptop.
> Having one tool that covers all the use cases just sounds like a very good way to increase the attack surface.
That’s just security by obscurity...
The real problem is that almost no one has been able to create alternatives to SystemD’s solutions to the vast majority of problems it chooses to tackle that are as well developed or effective as the SystemD solution.
You might then ask why no one created these better solutions 6 years ago before systemD gained its stranglehold on distributions, but that's like asking why systemD itself didn't add all these features 6 years ago.
Either open source and software freedom affords us endless amounts of choice and capability to create, inspect, modify and use whatever software we like, or we're all forced to use systemd against our will. Which is it?
Systemd doesn't have a stranglehold over anything - that misinterprets the power dynamic. It's got large adoption, because it's actually useful.
I don't think that's the case. Systemd was introduced with a SysV init backwards compatibility mode, so it was trivial to port services over to using it. Porting a systemd-only service over to SysV init is much harder, though. I'll be charitable and say that this one-way "interoperability" wasn't deliberate, but it does seem reminiscent of the way some proprietary software supports open standards for importing files but not exporting them.
> Either open source and software freedom affords us endless amounts of choice and capability
Software freedom is not a binary matter (excuse the pun), and different licences have given people different amounts of freedom in the past. For example, the FSF considered the pre-1999 BSD licence to be non-free simply because it required a sentence to be included in any advertisement of the product[0]. Another example would be "the Dissident test"[1] that Debian applies to licenses, considering them non-free if they require that modifications to the source code be reported back to the original author.
I think that when trying to determine the freeness of a piece of software, we have to consider the second-order effects it has on the free software ecosystem generally. If the adoption of a (hypothetical) piece of nominally free software caused other unrelated pieces of software to run artificially slower, or become insecure, or to no longer be developed, then that could be a net loss to the free software community, even if the new software has some shiny new features.
> It's got large adoption, because it's actually useful.
That's a rather self-serving interpretation of events. As another comment in this thread[2] explains, systemd at every stage of its adoption was made hard to avoid, and adopting it was made a one-way process. I also gave a more thorough analysis in a comment on another thread a couple of years ago[3], but it seems that the myths about systemd's path to world domination are becoming more and more accepted as people forget the actual history.
[0] https://www.gnu.org/licenses/bsd.html
[1] https://en.wikipedia.org/wiki/Debian_Free_Software_Guideline...
(I meant before systemd/upstart existed).
Is it harder because systemd supports more features than sysv? What's stopping someone writing a new init / service manager that's systemd-compatible? Or improving sysv so that it supports those features?
> Software freedom is not a binary matter (excuse the pun)...
Of course there's nuance, and you're right to point out that different corners of the open source world have different values and disagree on things (BSD/GNU etc). But the point is that the users are in control and that the power lies with the users. If systemd was bad, or actively useless, people would stop using redhat, Debian, etc. As proof that the overall model works, there are distributions out there that shun or provide non-systemd options. People can use and contribute to those!
I also don't agree that anything is a "one way process". Everything is temporary. We'll use systemd until something better comes along, just like we did with perl, apache, Hadoop, or any technology that was the de facto canonical way of doing things right up until the point that there were better options.
I remember the history too - systemd would have gotten zero traction anywhere if it wasn't actually solving real problems. Sure, there were some pretty major bugs, and I wish Debian had done it right, but that doesn't take away from the recognition that there was a problem to solve and this was our best stab at solving it.
I guess I just don't understand the energy poured into hyperbole like "world domination" and "stranglehold". If you want to convince me that something I find useful every single day is bad, then build something better.
I still don't think we were "locked into shell scripts" though. Even if shell scripts were the only thing that existed, it was still possible for someone to make a service manager which supported something else, with a backwards compatibility mode for shell scripts.
> What's stopping someone writing a new init / service manager that's systemd-compatible?
My understanding is that systemd deliberately makes it hard to independently reimplement certain critical components like logind and journald. In practice, no distro is going to go to the trouble of packaging and supporting a mixture of components from the "official" systemd set and other competing implementations, even if those competing implementations were easier to use, or faster, or more secure, or more portable.
Compare this to the situation where system services were not so tightly coupled and it was possible for distros to change their default ntpd, for example. Just as monopolies in industries prevent new innovative start-ups from emerging, a monoculture in Linux system services prevents people from even trying new approaches.
> As proof that the overall model works, there are distributions out there that shun or provide non-systemd options.
I can't help being reminded of the (in)famous scene of a Microsoft lawyer holding up a boxed copy of Red Hat as a defence against the claim that they were abusing their monopoly[0]. Alternatively, it's like an authoritarian dictatorship pointing to the existence of nominal opposition parties to show that its citizens are free.
My point is, I think you're missing some of the power dynamics here, in particular the fact that people don't want to completely reinstall their operating system just to avoid the problems of systemd. Indeed, most users wouldn't even know that systemd is the cause of their problems, especially when the problem is something nebulous like a lack of innovation and competition in the system services space.
To put it more simply: Just because lots of people don't switch away from Windows (or Facebook for example) doesn't mean that they are happy with it.
> systemd would have gotten zero traction anywhere if it wasn't actually solving real problems.
It might have solved a few people's real problems, but I think that GNOME suddenly making it a hard requirement probably mattered to more people.
> If you want to convince me that something I find useful every single day is bad, then build something better.
If you find it useful every day then I'm happy for you. However, I don't think I have to convince you of anything. I'm not trying to make it harder for you to use systemd, whereas systemd is making it harder for people to use other inits, including SysV init which has been working fine for them for decades. The onus should be on the ones doing the coercion, to convince the people whose choices they are reducing, not the other way around.
[0] https://archive.is/eMmMB / https://www.washingtonpost.com/wp-srv/business/longterm/micr...
(not sure how any of this is relevant to this discussion)
That is absolutely not the reason for systemd to exist, it is an additional benefit. How was early-boot logging, error handling, dependency handling between services implemented before? At best only in some ad-hoc way, different on each distro. The domain has an inherent complexity to it, init systems are not easy, there is no saving that. But I think systemd exposing the configurable parts in a declarative way is a net win over the previous solutions.
> That is absolutely not the reason for systemd to exist
If you read the original web document announcing systemd, one could come away with the impression that it was though (even if substantiated with quite questionable performance tests using virtual machines).
As a non-sysadmin, I just feel like it's totally fair to have PID 1 be about as trusted and uniform as the kernel. If there's a bug in systemd, that's bad. If there's a bug in Linux, that's bad. But can't we at least aim for an init that is not just a bunch of Bash scripts running as root?
IMHO, we should aim for an open standard for the init subsystem instead. For example, I want to implement my init system in Rust, but systemd is blob, which is hard to replace.
If I had to work in either day-to-day, I'd chew my hands off. :p
> IMHO, we should aim for an open standard for the init subsystem instead. For example, I want to implement my init system in Rust, but systemd is blob, which is hard to replace.
Well, hold on. The person I replied to was talking about being able to debug systemd. So this is very tangential to that. If we had an standard for inits, with different implementations, it would be hard to debug all of the ones that are compiled binaries. My point is still that a sysadmin should not be expected to debug PID 1 any more than they should be expected to debug a driver.
Also, you can just treat systemd's APIs as the standard. GNOME, for example, depends on launchd. And, IIRC, there was some shim created so that GNOME could run on a BSD, which obviously doesn't have systemd. So, go ahead and write your init in Rust- Rust has great C API/ABI compatibility!
https://github.com/mesalock-linux/minit/blob/master/src/main...
http://git.suckless.org/sinit/file/sinit.c.html
However, systemd is huge: 1.3 million lines of code.
https://www.theregister.com/2020/01/06/linux_2020_kernel_sys...
You're not even comparing them to just the init part of systemd, but the entire repository.
What's the point here?
It's not a job of init daemon to manage containers or resolve DNS names.
IMHO, systemd must be PID 2+, not PID 1.
Well, those initds don't actually start processes, they start one other process.
> It's not a job of init daemon to manage containers or resolve DNS names.
systemd's init doesn't do these things, other parts of systemd do that.
> IMHO, systemd must be PID 2+, not PID 1.
It is, for the most part (certainly most of those '1 million lines of code' are not in PID 1). Certainly you could try to keep more of the service management code out of PID 1, but it's honestly not something with huge practical benefits (your service management is going to be priviledged anyway, in the event of a crash all a non-PID 1 solution allows is maybe a little bit of an easier time giving a chance to inspect the wreckage before you need to restart everything anyway), and you create more interfaces in your system to manage.
systemd-as-init was fine. systemd replacing udevd, syslog†, mount, ntpd, DNS, dhcpd, and probably a bunch more stuff I've forgotten or don't know about, is something else. If I wanted a tightly-coupled opaque binary blob of a system I'd run Windows.
† Of course its log file is not ACID, so when it poops its pants and corrupts the file there's no way to recover it besides moving it out of the way. Though the log entries on why this happened are also corrupted so there's no root cause analysis.‡ They could have just used SQLite as the logging format and it would have probably been awesome with regards to tooling and doing queries (instead of grep-foo). But no.
‡ And journald also cannot send logs off-host in any useful format, so I end up having to run rsyslogd anyway to get things into my SIEM through something useful like RFC 3164 or 5424.
But yeah: unit files are (non-sarcastically) pretty good.
I also have a problem with socket activation. There is a reason why (x)inetd fell out of favor - its whole purpose was to provide socket activation. It's a hack that can produce new error states and it can actually slow down startup compared to starting services proactively.
That said, the OOM killer daemon seems to be a good thing.
It would be hard to argue, that systemd-resolved is worse/more-limited/more error prone/buggy than dnsmasq.
The socket activation is intended for services, that are not normally needed, but occasionally they might be. It doesn't apply much to servers serving well-defined services, but for desktop, it is a godsend (together with dbus-activation). Just because you don't have use for it for your use cases doesn't mean it is not useful for other people.
About the DNS service, I remember a bug where the fallback behavior was not what people wanted (it was something that seemed to be a good idea to systemd developers but didn't work well in completely reasonable real-world configurations), and the developers rejected the idea that systemd's behavior was wrong. Can't find it right now. I have disabled it on my system as well due to a problem I don't remember. Maybe it didn't use my router's DNS server and went straight to the internet. Maybe I just wanted it to leave my resolv.conf alone.
It was a fallback DNS used when the user has configured nothing.
Not only that, it was a default fallback that can be changed by both the distribution package and the user (including disabling completely).
It was a tempest in the teapot, by people who have to find some drama.
I don't remember that (not saying that it didn't happen, after a quick search I didn't find any filed issue, #18769 could in theory have similar symptoms), but I remember several threads, including LWN article about the existence of the global fallback.
This, naturally, just makes everything harder to have a sensible conversation about.
It's a breaking change for some users. I remember experiencing that bug.
I'll take up that argument. On Ubuntu 14.04 (and I think also 16.04), systemd-resolved used to crash on my desktop machine after about a week of uptime. I never figured out the reason; I got in the habit of manually pointing `/etc/resolv.conf` at a real server whenever it happened.
I don't remember ever having that problem with dnsmasq, but I've seen so many DNS proxy things over the years that I can't keep them all straight. I think it's probably fair to compare systemd-resolved and nscd; both were there by default doing something with DNS, and both crashed often enough to annoy me.
It's not a new problem, either. I remember whatever came with RedHat or Mandrake or whatever I was running circa 2001 also seemed to stop working after a while. I ended up writing my own DNS proxy in C++ and ended up learning a lot. The protocol is wildly different enough from almost any other common protocol that I'm not surprised when software gets it wrong (I sure did!). The takeaway for me is that designing a protocol is as much a human problem as it is a technical one.
Linux... Absolute nightmare. Every system had a slightly different DNS stack and trying to get queries to route over the VPN interface was an acrobatic exercise in docs navigation and trial and error. Ultimately, newer desktops with systemd-resolved were the easiest
systemd-resolved supporting DNSSEC is also a big win!
Tell me, a naive one, why a throwaway for this?
Damning with faint praise? :) While an uber-bin would be a bigger problem, there is a reason why I italicized "tightly-coupled". Often many people simply respond with "but there are multiple binaries",† which kind of misses the point.
> Tell me, a naive one, why a throwaway for this?
It was created 2017-01-01.
† Edit: Case in point, see sibling comment: "These are separate services, with separate binaries, with separate packages […]"
Exactly.
These are separate services, with separate binaries, with separate packages for most distributions and their use is optional; they have their uses. For example, most distributions won't use systemd-networkd, because it is (intentionally) quite limited and they use NetworkManager anyway. RHEL8 for example doesn't even ship a package systemd-networkd.
Then why are they in the same repo as the udevd code? And why did udevd have to be pulled into the same repo as the init code when it was doing just fine outside of it?
If you think they took the wrong approach and yours is the right one, show them by doing.
If Roger Ebert said a movie was awful, did people expect him to write, produce, direct, and/or act in one?
If Doug DeMuro said a car handles badly, are people expecting him to get a mechanical engineering degree and build a better one?
Since throw0101a has exactly zero public record of understanding the topic at hand, and everything he provided is an opinion, why should be that opinion taken into account? What exactly is his point, except for demonstrating that throw0101a doesn't like the existing approach?
The show them better is one way to establish your credentials that you know what you are talking about. There are other ways to achieve similar effect, but peanut gallery isn't it.
About the same as any other rando's on the Internet. Take it or leave it. Up or down vote me.
As I type this: I have 23 imaginary Internet points on the post that kicked-off this sub-thread, and 0 points on a post bringing up Ebert and DeMuro.
Whatever.
I'm neither down nor up voting you; I consider doing that to people I discuss with a bad form (yes, that's opinion too).
However, it helps the discussion if there are arguments for the positions of those discussing; the point is not for either side to "win" (whatever that means and however you measure that "win"), but to find the best outcome after considering all valid arguments.
However, that does not work when there aren't any arguments. Vaguely liking or not liking something, and having no idea about the
That's what Bitcoin is and people assign value to those useless bits don't they?
What's the point of life if you can't fret over magnetic ones and zeros?
Quite difficult to do with imaginary internet points tho ;)
There's a market for established Reddit accounts:
* https://blog.usejournal.com/what-i-learned-selling-my-reddit...
Don't get me wrong, systemd-networkd is fine for server or static usage; NM is better for desktop/laptop use.
For other services, such as journald and udevd, the dependency also works in reverse and the service manager can't run without the services.
In many cases, it's not entirely clear why they depend on the service manager itself. logind in particular was moved into systemd “in anticipation of the single writer cgroup architecture", an architecture that never came to be because it was quite clearly a very bad idea, and then elogind was forked to separate it again with no loss of functionality.
There is no reason for logind to exist, elogind should be the only thing that exists as it does the same thing without depending on systemd; elogind can also be used in conjunction with sytemd with no loss of functionality.
Even stranger things happen, such as DBus performing activation viā systemd by use of a private, nonsstandard a.p.i. for which there are standardized protocols specified by Linux-base that systemd also supports. That they chose to use a specific unstable, undocumented a.p.i. rather than a standardized one to do this is certainly a political rather than technical decision to create a dependency for it's own sake.
And that is indeed what many RedHat projects have done over the years; they have created dependencies on each other of little to no technical merit as form of product tying to encourage adoption of more RedHat software. — this is certainly not limited to sytemd.
I'm not an expert. I'm just a software dev and free software nut who wants a free software OS.
But (serious question), why does it matter that systemd has an ntp daemon or that it subsumed udev?
You say, disparagingly, that it is tightly-coupled and opaque (binary).
Why is loosely coupled better in these cases specifically?
Weren't `mount`, `ntpd`, `udev`, etc all compiled C code already? How does systmed make those service more opaque?
Everyone seems to hate journald. I'll go ahead and join in the chorus and say that it sounds really bad. But do we have to throw the baby out with the bath water? What else about systemd is actually worse than what we had before?
What does a time system have to with system start infrastructure? If you want to write a time daemon, write a time daemon: the chrony folks did, and it's pretty good. Lots of folks have switch away from the old school NTPd to it.
But it stood on its own merits, was able to mature over time, and when people thought it good, they could willing start using it (or not).
> Everyone seems to hate journald. I'll go ahead and join in the chorus and say that it sounds really bad. But do we have to throw the baby out with the bath water?
And that is exactly my point about things being tightly-coupled. If I (rightly or wrongly) perceive journald to be garbage, how do I not-use it?
Accurate time is vital for many cryptographic operations. For example, in the embedded space you may want to have a check that your current disk image (ideally checked by dm-verity) is actually up to date. And to verify the metadata signing that disk image using Uptane/TUF/SUIT or similar protocols you need verifiable time. Or you might need to contact a provisioning server at boot time, and need to be able to verify that server's TLS cert.
(Now, systemd used to have a bug[1] where time-sync.target would be reached before time was actually synced, and Poettering's response to the bug was the all-too-typical typical "yeah, don't worry about it, it's intended behaviour that the time-sync target doesn't mean that time is synced"[2]. But that did get fixed a couple years ago [3].)
[1] https://github.com/systemd/systemd/issues/5097 [2] https://github.com/systemd/systemd/issues/5097#issuecomment-... [3] https://github.com/systemd/systemd/pull/8494
He also suggests what they can add for people who want this behaviour.
Oh woe is me that systemd does not support my use case of an embedded system needing to verify disk image with accurate time before booting further, as the _default_.
> What does a time system have to with system start infrastructure? If you want to write a time daemon, write a time daemon: the chrony folks did, and it's pretty good. Lots of folks have switch away from the old school NTPd to it.
Nothing, but you're making the common mistake of thinking that systemd is an init system. It isn't. It's a suite of components that more or less can be enabled independently, that no-one's forcing anyone to use. One of these components is an init. Another one in a time sync daemon, etc. etc.
Chrony's great, but if you don't need most of its pretty advanced features, why bother? A simpler, smaller, 'good enough' client might be more suitable.
Isn't software choice a good thing?
Yes, but tightly coupling can reduce the ability to make a choice.
I personally need the features of chrony, so despite running debian stable / systemd everywhere, I just turn systemd-timesyncd off.
Tight coupling would be "you can't boot your system unless you use this time sync daemon implementation".
* The freedom to not run any program you do not wish to run, for any reason (freedom -1).
Presumably Stallman would agree that government-mandated spyware which happened to be GPL licensed would not really be granting users much freedom. The tight coupling of "you can't boot your system unless you use this time sync daemon implementation" might also count as a violation of this hypothetical freedom.
Applying this to systemd is more complicated though. I'm sure some people would say "Well it's released under a Free Software licence so you can remove or replace any part of the code you don't want.", but that seems almost as unhelpful as saying "Microsoft Windows is Free Software because you can always create a clean-room reimplementation of it".
[0] https://en.wikipedia.org/wiki/The_Free_Software_Definition
But my question was why does this matter? As far as I know, you don't have to use systemd's ntp thing. You might disagree with the project vision of systemd wanting to include an ntp daemon, but it sounds like it hasn't cause you or the chrony people any real harm. There was a good chance that, if systemd didn't have an ntp daemon, you'd have to replace your distro's default with chrony anyway. So just do that here, too, no?
> And that is exactly my point about things being tightly-coupled. If I (rightly or wrongly) perceive journald to be garbage, how do I not-use it?
You can't, AFAIK. And that sucks. I know you can disable its on-disk logs and enabled some other syslog service, though. In what way(s) is that insufficient?
Are there any other example of tight coupling that are bothersome in systemd?
> You say, disparagingly, that it is tightly-coupled and opaque (binary).
> Why is loosely coupled better in these cases specifically?
It creates extra work for others.
udev is far older than systemd; maintainership of the former was eventually inherited by one the latter's lead maintainers, who decided to move the latter into the former, at first promising that the former, which has applications reaching far outside systemd, could still be built and used independently.
That promise weakened after about two years and it became increasingly difficult to build and use udev without systemd; — some embedded systems that use udev can't even fit systemd into memory and thus using it is not an option.
So, udev in response was forked to eudev by other developers who now have to spend time and money on maintaining this fork and porting udev changes to it.
Loosely coupled is better because it allows one to be use without the other if there be such a need; tight coupling is essentially a form of product tying.
Why is it better that one can use an iPhone on any computer? rather than only computers designed by Apple?
> Everyone seems to hate journald. I'll go ahead and join in the chorus and say that it sounds really bad. But do we have to throw the baby out with the bath water? What else about systemd is actually worse than what we had before?
Many call for logind to replace acpid; — I strongly object to this and the difference in functionality between both highlights well the difference in philosophy between traditional Unix design and Lennart's brand.
acpid responds to acpi events by calling an executable file with a specific path; for instance `/etc/acpi/actions/powerbtn.sh` is called with arguments that describe the nature of the event when the power button is pressed. logind on the other hand simply allows for a limited list of I believe seven options in a configuration file of what to do when the power button is pressed, not an arbitrary file to be executed.
The possibilities of acpid are obviously limitless, and I use them as such. My machine is configured such that when I close my notebook's lid, the machine does not suspend, but rather disables the screen and goes into power-safe mode, but otherwise continues to operate, I can override this behavior by simply creating the file `/run/lid.disable`.
I cannot configure my machine to be so flexible with logind which highlights the difference in culture: Traditional Unix design was always about removing restrictions, and this is about providing features, but when the feature one seeks not be provided, then one is out of luck.
When current journals are detected as dirty by journald @ startup, they are renamed and no longer written to. This is a very robust and conservative approach. It burns some space in the interest of simplicity and preserving the log data without risk of messing things up.
These renamed journals are still accessed when reading, so it's not like those dirty files no longer participate in journalctl operations; their contents are not lost.
Any logs lost in a crash are just what was in-flight IPC, or sitting dirty in the kernel's buffer cache and hadn't been sync'd to backing store yet. Journald also performs an explicit sync whenever an urgent message arrives, to try ensure it's made durable ASAP. Of course there's still a filesystem and potentially myriad layers below that which could lose data in a crash.
ACID would be overkill for this fairly simple single-writer multiple-reader mostly-appended situation.
Edit:
Your statement "when it poops its pants and corrupts the file" implies journald has a tendency to actively corrupt its own files.
The principal mechanism journald uses to detect potentially corrupt files is simply identifying a file as being ONLINE when opening for writing, typically at startup. This is almost always caused by an unclean shutdown of the host, though it could be due to a crash of journald itself.
The file isn't even necessarily corrupt. It's simply assumed inconsistent and treated as such WRT writability. I'm not aware of any journald bugs in recent history where journald actively corrupted the contents of journal files. Even crashy bugs where journald was littering dirty journals to be treated as corrupt I'm not aware of being actually corrupted internally by journald bugs.
Source: I've worked quite a bit on journald over the years
I personally like service managers more that have clean shell scripts such as runit or OpenRC where shell scripts more so look like this:
#!/usr/bin/openrc-run
depends="some list of services to depend on"
after="some list of services which need to start earlier"
cmd="/path/to/executable"
args="command line arguments"
I actually like journald quite a bit. Before, you'd have log files scattered all over the place with various cronjobs to try to keep them from filling up the disk (usually with the help of logrotate). Each application was free to put its logs wherever it decided.
Journald keeps them structured with first-class support for metadata like timestamp and unit so it's easy to get a service's logs in a 2 hour window on a given day and equally as easy to look at them in a stream with other service's logs (it aggregates them for you). From an application perspective, you just dump to stdout/err.
>And journald also cannot send logs off-host in any useful format
This seems directly at odds with your "tightly-coupled" beginning statement (you want it to do more or you think it does too much?). It's trivial to do this with fluentd or other log aggregation software
Those shell scripts never had to be cumbersome. The BSDs, for example, do away with a lot of that complexity and boilerplate quite nicely via e.g. rc.subr and rc.conf.
Basically change aversion, and a misguided idea about the "unix way".
I have done most commercial UNIXes starting with Xenix in 1993, none of them was (or are, regarding the surviving ones) following it.
In fact some of them have introduced systemd like ideas way longer.
Also modules, objects and components packing functions together as means to make sense of disjointed functions.
True enough. I spent a fair amount of time trying to wrangle with smit on AIX. That's not an endorsement :)
(1) It's a "god object" at the system level, rolling massive amounts of functionality into a single monolithic "system manager." There are advantages to that, but it also makes it impossible to strip away complexity and leads to the core system becoming really bloated with functionality. That's the road Windows took that made Windows such a mess of cruft.
(2) Poor UX... it's obtuse and counter-intuitive. This is my biggest gripe. It replaced a hard to use ball of shell scripts with... something even more confusing! Every command seems designed to be hard to remember and clunky to type. Common tasks are "undiscoverable" in that you have to search docs to figure out how to do the simplest things. Git has this same problem by the way, but to a somewhat lesser degree.
On a system where the primary API is the shell, initialising everything with a sequence of imperative shell commands makes a lot of sense.
Getting it right in serial fashion is fairly easy, but since systemd insists on doing as much as possible in parallel you really need to have every dependency correctly configured (which is not always that easy to get right if things take a couple of seconds to get going..).
Either that is missing or you have some other config error, like the service believing it is ready, because it is not told about its state correctly?
But there is a setting to fall back to serial mode I believe, but it will also not work correctly if your service files are configured incorrectly.
It is much less forgiving to mistakes (and yes.. the little annoyance I am having right now has the dependencies set correctly, correct service type, systemd-analyze chain looks good.. so mistake is somewhat well hidden).
One of the (very many) things that gave me and my colleagues a headache when switching to Systemd was two init scripts that could run one after the other (and the order didn't matter) but they could not run simultaneously. This had worked fine with System V style init scripts that ran serially, but started breaking in Systemd.
Of course you can now argue that this is a configuration error - however, it does not change the fact there there didn't use to be a race condition. Now there was one. One is a larger number than zero.
In the short term, things breaking like this are of course frustrating, but if it was my environment I’d prefer that this dependency was an explicit part of the configuration rather than something that works “by accident”, unless one prefers that initialization should never be anything but serial, but I personally at least don’t see the benefits of that outweighing what systemd offers.
Not really. That wasn't the question though. The question was "were there more race condition with the parallel system than with the serial system?". The answer was undeniably, objectively yes.
Nonetheless, parallelism is an additional feature (in my opinion, greatly appreciated), but it can be turned off. Additional functionality is rarely bad over not having it.
They configured firewall rules using iptables. I'm not sure I consider configuring firewall rules at startup using an init script "really really shady". Turns out (quite unsurprisingly) that you can't run multiple instances of the iptables command simultaneously.
I don't really want to get into the whole debate about whether parallelizing startup scripts is better or not. I'm not saying it's not, but that's beside the point. The point was that parallelizing startup scripts leads to a set of problems that didn't exist before. It's annoying to see people argue that those problems don't exist. It's like arguing that the problem of flat tyres aren't more common for people driving cars than they are for people riding horses. It's utter nonsense, no matter how many other advantages cars may have.
Also, are you sure it is a race condition? Some other errors can also appear as races. What service are you trying to create?
And my problem is the very definition of a race-conditions. It doesn't always occur and if I play with the timing a bit I can make it go away. The dependency unit is a oneshot script that adds some ip addresses to interfaces. The dependent job tries to bind to one of those ip addresses. And yes the interface is up before this all happens.
It would’ve been cool to have functions that supported dependency in the scripts themselves. A web server would have a call to...
init_wait_for networking
init_ready networking
Doing all this with INI files is very clever but it’s always felt very NIH. Good solutions solve existing problems without rewriting and embracing everything from the ground up.
That being said, I really wish there was more interest from the systemd folks in interoperating with containers and registries and all things OCI in general. Like the portable services they introduced in 239, the storage for the system extensions introduced in 248 is backed by either a directory or a disk image; meanwhile, there's an entire ecosystem of tools in the container world dedicated to the storage and distribution of root filesystems that they seem to be completely ignoring. I know there's nothing stopping me from using `umoci` or some such to pull and unpack a container and use the resulting directory as either a system extension or a source of portable services, but it'd be a lot cooler if that kind of functionality was baked in.
No argument there (even though declarative programming languages aren't universally favored over imperative ones), but why does this come up so often? On a typical Linux desktop or server there are hundreds, if not thousands of packages installed, yet only a few dozen init scripts. Almost no tool or application is typically run as service requiring such.
Just the dependency resolution features alone make it fantastic. It's only a few lines to specify a network dependency, a filesystem dependency, another service dependency, timeout and retry behaviors, monitoring and alerting on failures, etc.
Plus it goes beyond just init. The scheduling features mean I'm turning to it instead of something like cron for doing repeated tasks on a server. Again, there's just a bunch of quality-of-life improvements there, like dependencies, better and more granular logging, error reporting, retries with limits, etc.
I build embedded systems and appliances; it sounds like you might have to deal with this a lot less frequently than I do. I don't have to deal with it as frequently in the age of systemd though, because more projects are providing unit files now that it's everywhere, and it's a lot easier to glance at a systemd unit someone else wrote and decide if it's correct than it is to audit a random shell script and figure out what needs to be added or changed to get it to work correctly in your distro of choice.
That'll be the case. When I did work on an embedded system (enterprise storage system) different groups worked on different services (and some used hair-rising shell scripts to start their service), but our team's service was started via xinetd and we didn't spend all that much time on that aspect.
Yes you can argue it's the distros decision, but that ship has sailed and fairly soon there won't even be an effective choice.
A bit of a niche use perhaps, but seems to be very useful to me. And it's good that it's in systemd because otherwise you'd have to hack the boot process of every distro you might want this in.
And I wouldn't say it wasn't broken, it's just a need you personally didn't have. I do actually have this need in a virtualization context, and may well making use of it.
Also, FreeBSD wouldn't work. We use many Linux specific features, and I have no interest whatsoever in porting all that stuff to BSD for no gain in functionality.
Also, I see no issue to somebody else solving my problems and not needing to write more stuff that's unrelated to the actual purpose of what I'm doing.
Is this not a fact, versus a mere argument?
Do you think the decision is not based on merit? b/c there's all the effective distros switching, versus you saying nothing is broken - one of you must be wrong?
Maybe the Linux ecosystem is broken, you just can't say it because you've already adjusted you expectations, or solidified your use case(s)?
Honestly the whole thing made me more or less give up on open-source as a development model. Good programmers are expensive, especially if you want them to do the boring maintenance work, and that funding has to come from somewhere. The "give the program away for free and sell consulting" business model works up to a point, but it distorts maintainers' interests away from what would actually benefit the majority of users. While I do blame RedHat, the real problem is that no-one except RedHat is paying people to work full-time on GNOME or any of dozens of similarly important projects (KDE seems to be mostly funded by the EU, which is a much healthier model, but I don't know if you could make a whole distribution from projects like that).
Having been born during 70's I also saw all those ideals turn into business people the following decade.
It is easy to be idealistic when others put in the money.
How is it anything like Windows? You're honestly, without bad faith, comparing it to a closed source operating systems that has telemetry enabled by default and forces you to run system updates?
I have no idea what systemd-oomd is, but from a cursory reading of the article I can tell that it's a .service, and from the description that it's a "userspace out-of-memory (OOM) killer". So either this service will (worst-case) be enabled by default, in which case I presume that you can disable it or mask it completely like any service, or (best-case) it will be disabled by default, in which case I really don't understand why it matters.
In addition to systemd fully allowing you to disable and re-configure much of its functionality, it's also fully open source. You can re-build it however you want, if you want to use it in another manner. This is what passes for being reminiscent of Windows?
And as the scope continues to increase, you're still either all in or all out.
As the distros fall into line, you're either all in or all out.
Try and not use systemd, and then tell me it's not a problem. I think it's honestly easier to not use windows at this stage. The BSDs (love them as I do) just don't have the clout or the support for many day to day developer needs.
This is not the case with systemd. Although these projects live in the same repository, distributions can pick and choose them depending on your priorities.
Granted, this is obviously not something most end users come in contact with and even know since very few people actually build distributions. As an end user, ripping out a core part of a distro to replace it is always a painful experience. Regardless of whether that component lives in the systemd repo or somewhere else. Systemd actually makes ripping out core parts easier, which is why arch adopted it so quickly, but that's another story.
You're just throwing vague and incorrect statements out there, sprinkle "Windows" on top in the hopes of eliciting an emotional response.
* systemd-cryptsetup gained support for unlocking LUKS2 volumes using
TPM2 hardware, as well as FIDO2 security tokens (in addition to the
pre-existing support for PKCS#11 security tokens).
Granted, in case of theft, the computer will boot up if nothing is done to it. But in the case of a "user" computer where no services are listening, the risk is relatively low. In order to get access to the decrypted disk contents you'd have to exploit some vulnerability in Linux and / or the login manager.
Some people consider it more convenient to just boot the thing, take the protection against casual loss of confidentiality and accept the risk they might face from determined attackers.
This is a tradeoff. A "random user"[0] might be put off by having to type in a password every time they boot up their PC and forego disk encryption altogether. This approach is a net improvement over an unencrypted drive, even if it's not perfect.
As usual in matters of security, threat modelling is important. Who are you and who are your defending againts?
Are you some high(-enough)-profile target with potential access to interesting infrastructure or data? You may want to not rely on only this approach. As siblings have said, some kind of supplemental factor could be better. Say a challenge-response factor based on a Yubikey. Also, if the thief is determined, they may attempt to steal the laptop while it's running. You may want to look into desktops if that's the case. And not plug them into a UPS, of course.
But are you just a random Joe whose threat model is that you have some photos you wouldn't like seen, maybe some personal documents, but not much more? Then you need "just enough protection". If someone steals your PC, it's likely for the PC itself and not for the data it carries. They may want to check what's on it, see if there's anything juicy. Might even try "password" or something to unlock it. But they will probably not spend too much time trying to brute force Linux (or Windows for that matter).
> Almost every machine has some services exposed
How often is this the case on a random user's machine?
Even on my Windows machine I've only got SMB and RDP listening. Which I've manually enabled.
On my Arch Linux machine, which is my actual work machine, only SSH listens. Which I've manually installed and enabled. On an ubuntu desktop 20.10 install, there's no SSH server installed. There are no daemons listening on the external interface.
I guess you may attempt some attack against the DHCP client if you really want to get into that machine. I also suppose the pipe wrench exploit can still be tried.
> and there are also hardware-based attacks possible
Don't you think that if someone's going to invest the time and ressources to conduct such an attack against your PC you might be the kind of target who should be aware that's a possibility? Which brings us back to the point above about threat-modelling.
---
[0] I realize we're talking about LUKS and Linux, so maybe the "random user" isn't all that random. But maybe it's a "mom's computer" on which someone installed Ubuntu or something.
People need to be aware this can happen when using it and should remember to add a password key to their luks container.
Though I'm fairly certain that Linux will forego this and just assumes you known what you're doing.
Yes, even the documentation for the systemd feature implies generating the recovery key is a separate optional step.
That has been possible (without TPM) since about 2009: Shameless plug: https://www.recompile.se/mandos
I'm gonna try it out once my Solo V2 arrives.
Somebody has got to stop systemd from unilaterally fucking up every Linux distribution's design with their wacky features. Only /opt and /usr? Because no other directories might need/want an overlay? And why the fuck can't we just use a regular overlay fs? To say nothing of it actually breaking the expectations of users and operators ("In addition to your existing huge troubleshooting checklist, see if systemd is silently injecting fake files into your process's path")
When the hostname is set explicitly to "localhost", systemd-hostnamed
will respect this. Previously such a setting would be mostly silently
ignored.
This whole thing is designed like Kubernetes. Come up with a bad design, then hack in random functionality to make it as complicated as possible to use.
systemd-networkd, systemd-resolved, systemd-timesyncd, systemd-udevd, systemd-journald, systemd-logind, systemd-nspawn, systemd-cron, systemd-notify, systemd-container, systemd-boot, systemd-oomd, systemd-mount, dbus, ...
and oh good --json logging support is available everywhere. nice SYSTEMD_COLORS supports new 16 and 256 options for all these tools. my entire system will work much better. with high consistency.
this whole monorepo (not monolith) thing is working out.
I understand it's not a good fit in some cases; our standard server config disables all other NTP clients, systemd-timesyncd included, in favor of Chrony because we are concerned about highly accurate system clocks and slewing. Interestingly, it appears that systemd has some sort of provisions for this. The `timedatectl` command still works and reports that NTP is enabled and synchronized.
How so? Compare what msntp does to adjust the time with what systemd-timesync does. Though msntp does have an "immediate" option if you want to be abrupt.
There's a bunch of discussion of systemd-oomd and a bit of discussion of systemd extension images. Much of it is even calm and technical.
Edit: I guess if you have multiple services from multiple vendors that nevertheless agree to some common base image, it makes sense for those services to use `ExtensionImages`. Eg multiple services that all require a libglib base image. A Portable Service can expose multiple services from a single disk image, but that's the point - the vendors have to build a single disk image with their services.
[1]: https://github.com/systemd/systemd/blob/main/docs/PORTABLE_S...
I sometimes wonder why we even install stuff - mount it it in from remote with some kind of caching system, and you'd always have the latest, like on the web. Slap a delta algorithm on top, and it could even be more efficient.
This is not to say that there's no value in freezing stuff and staying in control - there is - but ideally we would be able to choose.
Think of them as the "app" part similar to mobile OSes; fedora silverblue/rpm-ostree/systemd-extensions as the "system" part.
[1]: http://0pointer.net/blog/projects/stateless.html
[2]: http://0pointer.net/blog/revisiting-how-we-put-together-linu...
[3]: https://www.freedesktop.org/software/systemd/man/sysusers.d....
And I agree: one could use systemd-sysext with something like rpm-ostree to create very robust systems with selective immutability.
https://www.freedesktop.org/software/systemd/man/systemd-sys...
Anyway, I reported it and it presumably got fixed. I wouldn't know, because systemd dev handling this didn't say a thing to me at all. Just closed the report after some seemingly not very related commits to what I saw in gdb.
I just dropped the pam module from config anyway, and I'm not willing to lose access to my cluster again just to try.
Not sure how distro fucked up in this case, when it's homed pam module going into infinite loop once in a while unpredictably blocking all processes that use pam naively, unless you mean that they fucked up by using systemd or pam at all.
Edit: lol, I read through systemd release notes and there's no mention of new huge systemd-homed pam module at all. But good I looked, they're moving pam config for systemd to /usr/lib/pam.d and systemd-home pam module is again enabled there. So I'll have to check again all my systems if it's really disabled after upgrade.
Is there a document somewhere that shows what the systemd team thinks is a functional boundary? What things they don't plan on replacing?
1. makes very common usecases needlessly hard for beginners (e.g. run something on boot)
2. is sometimes the opposite of self explainatory
I use systemd and am OK with it and I aggree with the problems it is trying to solve. It is just a bit convoluted and intransparent even after years of using it.
And if you just want to start a gui, Gnome Tweaks has a page which is super simple.
Ah yes, systemd unit files are well regarded as far more complex than shell scripts that are 99% boilerplate. /s
Actually, unit files are possibly the only part of SystemD that I've knowingly interacted with - I know it does about a thousand things, but whatever those other things are, they are transparent to me as a user.
The biggest problem I remember now is that when testing your init script it picks up environment variables from your root shell. So, because of that it works when it shouldn't, and realizing what's missing is a big debugging nightmare. The systemd init starts every unit in the same environment.
Another one that resulted in hours of fun fixing it is init scripts happily starting services even when various filesystems failed to mount. Like /var. Getting those particular database files back in sync after the / filesystem filled up was a nightmare.
Systemd has pretty good documentation and I think it is a feature that its users are pushed to read it, rather than hacking out a shell script that doesn't use half the already existing functions because the author never bothered to read RedHat or Debian docs.
...the 90s. So it's too late to change them. We should definitely be considering safer languages in 2021.
People all the time complain about systemd bloat, but it's very likely that in those languages it would be far worse.