Hacker News Reader

Comment by bjackman | Hacker News Reader

He's referring to the fact that systemd was (is) very controversial when it was announced as a replacement for SystemV init. There were _lots_ of flamewars and hyperbole and when Debian announced they were making systemd their only init, it sparked a fork: http://debianfork.org/

It seems like a lot of people are calming down about the issue now.. so lets not do all that again

View on HN

I think Linus Torvalds coming out in support of systemd helped a lot (or at least confused the UNIX purists enough to calm them down a little). Linus response from [0]:

> I have to say, I don't really get the hatred of systemd. I think it improves a lot on the state of init, and no, I don't see myself getting into that whole area.

> Yeah, it may have a few odd corners here and there, and I'm sure you'll find things to despise. That happens in every project. I'm not a huge fan of the binary logging, for example. But that's just an example. I much prefer systemd's infrastructure for starting services over traditional init, and I think that's a much bigger design decision.

> Yeah, I've had some personality issues with some of the maintainers, but that's about how you handle bug reports and accept blame (or not) for when things go wrong. If people thought that meant that I dislike systemd, I will have to disappoint you guys.

[0] http://linux.slashdot.org/story/15/06/30/0058243/interviews-...

I'm not sure that Torvalds says that he likes systemd. I read the quote as saying: "I don't dislike systemd, and I think that it is better than traditional init.". Almost anything [0] is better than SysV init.

shrug

[0] Upstart, OpenRC, daemontools...

Strictly speaking, of those three only upstart is actually a replacement for System 5 init. OpenRC and daemontools run under an init, and by and large replace System 5 rc. There are two toolsets (soon to be three) in the daemontools family that provide a replacement for init: nosh and runit. daemontools itself does not, and has had more than a decade of people adapting it to run under init replacements because the vanilla system only targets late 1990s inits, including under upstart as a matter of fact.

* http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/da...

* http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/in...

* http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/sy...

So, strictly speaking, is SysV init more than what is described in init(8)? That is to say, is it more than

* the parent of all processes, including orphans

* the executer of various entries in /etc/inittab, depending on runlevel

* the configurer of local and serial consoles

* the handler of CTRL+ALT+DEL

See http://unix.stackexchange.com/a/197472/5132 as well as https://news.ycombinator.com/item?id=8907445 and https://news.ycombinator.com/item?id=8904429 .

From one pedant to another, thanks for the correction, and the informative links!

So, I guess that the Debian folks were incorrectly claiming that they were looking for a replacement init. They were looking for a replacement rc system, and were willing to accept a replacement init as part of the package.

Never mind that the term systemd can be used both for the init binary, and the larger project (systemd, networkd, udev, journald, logind, and the list goes on and on).

As best i could tell, Torvalds didn't have an issue with systemd as the init.

Nor do i think many would (outside of the whole journald thing), except for the snowballing of sub-daemons and absorbed projects (udev for instance existed for a decade as an independent project before being folded into the systemd code tree).

Without all that, it may well be that systemd would just be one init among many (i barely noticed the existence of upstart for instance).

I wasn't born yesterday, I know that. In fact, I helped fan the flames by making uselessd, though not out of any sly intent in the least, but research curiosity.

My question was more about the GP elaborating why a new system would be something "[no one] wants to go through this again."

(I'd characterize systemd as being a sysvinit replacement to be a great misnomer. It's not. It's an entire framework for providing certain low-level userspace daemons, utilities and auxiliaries meant as a common middleware for GNU/Linux distributions. Even systemd-the-PID1 cannot be called a sysvinit replacement in any integrity.)

>My question was more about the GP elaborating why a new system would be something "[no one] wants to go through this again."

Because the migration was terrible that is why. And I still continue to find stuff I hate about systemd. The latest was just yesterday, while troubleshooting some disk issues I wanted to fsck root before rebooting. Which normally means just remount root as read-only. Tried that and it didn't work, "disk was busy". Jumped into runlevel 1, still disk was busy. Rebooted into rescue mode, remount ro, still disk busy. After googling and digging through lsof, I could see it was systemd processes and journal that was holding onto files and preventing a remount. Tried killing systemd, journald, even kill -9, it just restarts it self and holds the disk. Darn. Did the touch /forcefsck trick, rebooted, yet no fsck. Darn again. More googling later, I see you have pass kernel parameters at boot time to force a fsck, and no, you can't select which FS either, it just does all of them. Rebooted again, fsck is running of course. I just wanted root checked but now it is doing all my terabyte partitions as well. Gave up at this point and just left the machine and hoped for the best.

I hate it too, but workaround is not too hard:

  chmod a-x journald && pkill journald

of course, it is bad idea to touch damaged FS before fsck.

Wait.

To run an unscheduled fsck of your root fs (and only your root fs), you have to remove the execute bit from a systemd component, kill that component, then remember to reset the bit after you're done?

That's nuts.

That's nothing - systemd panics (pid1 crash) if you restart dbus, which requires either the Magic SysRq or the reset button..

https://forums.gentoo.org/viewtopic-p-7811550.html#7811550

Pottering was apparently told about this in 2013, when he simply declared it expected behavior because dbus was "rock solid".

http://lists.freedesktop.org/archives/systemd-devel/2013-Jan...

Incidentally, this might be the original reason for kdbus - it's harder to restart the daemon after it becomes a kernel module.

> Incidentally, this might be the original reason for kdbus...

Yeah, especially given that its stated reason (Moving into the kernel gives us a 2x speedup over userspace dbus! Dbus is slow and we NEEED the speedup!) is mooted by the 10x speedup that cleaning up the userspace dbus libraries has achieved. [0]

Having said that, in the systemd-devel post Poettering says that they're working on...

> ...improving this in two ways: even if it still brings down the system, at least allow logind to handle this case nicely, so that you get a sane getty. [The other way is to push dbus into the kernel.]

Systemd is a sprawling project, so I could see why it has taken more than two-and-a-half years to fix a local DOS caused by the perfectly normal method of responding to a DBUS upgrade. ;)

[0] Fun fact: Those cleaned up libraries were supposed to be released after kdbus was merged into the kernel in 4.1. The Systemd Cabal was so disappointed that they had to ship the performant userspace dbus libs before the dbus daemon got pushed into kernelspace.

I am by no means knowledgable in the relevant domain yet this is scary at so many levels. The team that is taking charge of so many critical infrastructure not doing a minimal level of fact finding before pushing for a new kernel module for RPC is a bit obscene.

I may have read things that are biased but I have a feeling it is a useful project and it does a lot of thing elegantly or right. But the project acts like it is the center of the universe and all should act to meet its needs. I have seen such project/modules at work, they usually makes everyone happy to start with and down the road when they finally get stuck and I am forced to consult we are already long way down the slope.

Goddamn, guys, stop repeating the same absurd.

kdbus is NOT ONLY about performance. It's about the lifetimes, the availiability in the early userspace, the new marshalling format, the new user bus concept, and so on, and so forth. It's about separating the transport (in the kernel) and the policy (in the userspace).

Most of these points can be achieved without putting the transport in the kernel, but some can not.

Yeah right. No wonder I see people adding all sorts of adjective to the systemd proponent. I don't usually go hunting for sources and I only had Linus's comment about kdbus and performance. So I went to look for pottering's own word and here is a verbatime quote from the abstract of his presentation on kdbus

  D-Bus is a powerful design. However, being mostly a userspace solution its latency and throughput are not ideal. A full transaction consisting of method call and method reply requires 10 (!) copy operations for the messages passed. It is only useful for transfer of control messages, and not capable of streaming larger amounts of data.

So yeah he tried to sell performance as the big thing for his version of dbus IPC. I'm guessing he (and group) spent more time with dbus than linus and failed to find the actual cause of the performance issue or tried to hide it give more credence to kdbus. In both cases it makes it that much harder to trust their reasoning and conclusions.

I never said kdbus is ONLY about performance. But from what I'm reading they are not performance people but they think they are. The most dangerous people are those who don't know their own limitation or weakness. And your comment simply emphasise the practice of listening whats being said. So yeah given that I don't know dbus or ipc I have to trust your logic and you only make it hard to do so.

Seems the quote gets turned into a single long line in Firefox, and subsequently making the page extra wide...

Strange. The quote is properly contained over here.

FF 40.0.3 on 32-bit Linux.

If you haven't, you should go back and read all of the LKML discussion surrounding the request to merge kdbus into Linux 4.1. Perhaps I misunderstood something, but I got the distinct impression that several of the folks reviewing the code were concerned that kdbus pushes too much policy into the kernel.

LWN article: https://lwn.net/Articles/641275/

LKML thread: http://thread.gmane.org/gmane.linux.kernel/1930358

> It's about the lifetimes, the availiability in the early userspace...

Starting dbus in your initrd [0] makes it available to the system from before the first second your service start machinery is started. If you want dbus available to userspace in your initrd, make starting dbus the first thing you do after you load your initrd. Is there some particular reason why this doesn't meet the lifetime and availability requirements?

[0] Remember that the initrd is responsible for things like decrypting the rootfs. When you're in the initrd, your real system is often not even remotely ready to run.

That kind of thing is why I stick with Slackware and OpenBSD for my production systems. systemd has some great ideas, but it has some down right silliness as well, and I personally don't feel it's stable enough for daily use yet.

(Just to make sure everyone gets it)

It will prevent it from running again. So it is basically a big FU! to systemd.

"So you you think you gonna restart yourself? Oh, no you won't"

As mentioned above, that doesn't seem needed [ed: nm, it is - if journald is set to log somewhere other than /run that's usually mounted as a tmpfs].

[S]urely, something like:

  mount -t tmpfs /run && pkill journald

[ed: that is, for logs on /var/log: "mount -t tmpfs /var/log && pkill journald"] would make more sense? (In my experience systemd itself does something funky on / -- so the above wouldn't be enough -- but still seem a little more sane than running a chmod on a file (presumably) residing in an fs you want to fsck).

Actually, the workaround is to reboot with kernel param "emergency". It's single user mode with the root helpfully already in read-only mode.

The OS is Debian 8. I seemed to think the same too, but as I mentioned, rebooting into single user mode did no such thing.

You mentioned booting into rescue mode. Rescue mode is not emergency mode. Have a nearly two decades old explanation that still holds true for systems like nosh and systemd today:

> If you need to check the root file system, you can reboot into single-user mode with the root partition mounted read-only by issuing the -b switch at the LILO prompt. The -b switch will be passed through LILO to init and will cause an emergency boot ...

-- David A. Bandel (1997-01-01). Disk Maintenance under Linux. Linux Journal. http://linuxjournal.com/article/193

That points towards either Debian implementing systemd poorly in this aspect, Debian poorly documenting well how to accomplish this after the change, or a failure on the user's part to find said documentation.

Systemd on Fedora supports this perfectly fine, so I can't really call this a failure of systemd itself.

Maybe the problem is that you're carrying conventions from an obsolete system to a new one? A quick google search for how to force fsck with systemd comes up with results[1] mentioning options you haven't covered. Then again, I haven't had to do this, so maybe you did try those steps and it didn't work.

1: http://lmgtfy.com/?q=systemd+force+fsck

As far as I can see, they all mention "fsck.mode=force, as a kernel parameter", which is what I was complaining about.

Here's the TL;DR

1) It's actually super easy to get to a read-only root mode.

2) It actually is a problem of carrying over obsolete conventions from prior init systems.

3) It's not your fault because systemd has extremely poor documentation on this AFAICT, to the point that people searching for this problem can't even get answers from others that know how to do it, or at least those answers don't rate very high.

4) Just boot with kernel param "emergency" instead of "single" (or "rescue" which is an alias for "single") and you will get a single user mode with root already mounted read-only for you.

And now for the longer rambling version.

Okay, so I spent a little time on this since I happen to have a virt for Fedora 20, and I finally found a couple of solutions that work. First, the systemd solution, which is emergency mode. It's correct that the journal will cause problems here, but that's because the journal was already running and apparently systemd doesn't want to, or is unable to stop it. The solution is to reboot into emergency mode. This can be accomplished by using the kernel param "systemd.unit=emergency.target", or just the param "emergency". This will not only boot you into a single user mode, but will default to having root mounted read-only.

To clarify, this appears to be a new mode in addition to single user mode. Single user mode can still be reached with the kernel param "single" or "rescue" or "systemd.unit=rescue.target".

I finally stumbled on this when I decided to look at the directions for what I wanted to accomplish for the distro I was using[1] and not for systemd specifically. Which makes sense, but I think we are all to stuck on "systemd is different, how do I do this in systemd" rather than "how does my distro say I should accomplish this now, given the changes they've instituted."

1: https://docs.fedoraproject.org/en-US/Fedora/18/html/Installa...

2) It actually is a problem of carrying over obsolete conventions from prior init systems.

It actually is not. It's a problem of not knowing the conventions from prior init systems. emergency mode did not originate with systemd. It has been around since 1995-12-03 (Miquel van Smoorenburg's System 5 init clone, version 2.57d).

It was possible to boot to single user mode and remount root as read-only prior to systemd, Whether this was the optimal method for achieving this is irrelevant, it was possible, and popular. That specific method no longer works easily as before, while the optimal method does. The commentor was using their preferred (if suboptimal) convention from the prior init system and running into problems. I think that perfectly matches the statement "It actually is a problem of carrying over obsolete conventions from prior init systems."

It is also a case of not knowing the correct/optimal prior convention, but that doesn't make the original statement untrue.

This is a job for a distro like System Rescue CD. Boot off CD, run in DRAM, figure out which partition or LV is root, and directly fsck it.

[ed2: Never mind - the standard setup is to have /run mounted as a tmpfs -- I failed to check that before this song and dance]

Which distribution was this? I just tried on my laptop (running Debian 8, fs on top of lvm on top of LUKS), and a simple:

  telinit 1
  # log in on console
  mount -oro,remount /

worked as expected.

[ed: Actually just tried forcing an fsck as well, and worked without a hitch. Just remember to "mount -oremount /" before running a "telinit 5" (Nothing really bad happens if not, but the system(d) will be confused if rootfs is mounted read-only).]

esaym probably doesn't need people to try and diagnose xyr problem, and this isn't really the best venue to do that. If you want to try this, try at least replicating esaym's setup as described rather than using your own and declaring "works for me!". One way that springs to mind is moving the journal into /var/log and having no separate /var/log volume; which reproduces the described setup of open journal files on the root filesystem. Do that and then try to force an explicit check of the root volume.

Of course, knowing that does point to one obvious way to avoid the stated problem: move the journal back into /run again (per the journald.conf manual page) and simply restart journald. No mucking about with permissions required. But that's not the only thing that xe could have tried. emergency mode, for example, is documented as not mounting additional non-API filesystems at all, nor read-write remounting the root volume. See http://freedesktop.org/wiki/Software/systemd/Debugging/#boot...

Hm, good catch. I thought I checked to see if /run was mounted before going through the whole exercise, but apparently I didn't (My first test was to "mount -t tmpfs /run" -- to shadow the /run mountpoint -- but I didn't realize this already existed as a tmpfs).

[ed: Given that it's not wise to write to a fs of questionable state, editing /etc/systemd/journal.conf is out. So it would appear shadowing the sub-tree in question with a tmpfs mount would be the sanest approach?]

[ed2: Just checked that if setting "Storage=persistent" in journald.conf, forcing logging under /var/log (and with /var/log as part of the rootfs) at least here, shadowing /var/log with a tmpfs works, in order to remount rootfs read-only in "runlevel 1" (actually systemctl rescue, when using systemd). Not sure if there's any elegant way to unshadow/unmount /var/log w/o a reboot though.]

Reply on news.ycombinator.com