Hacker News Reader

I hate it too, but workaround is not too hard:

  chmod a-x journald && pkill journald

of course, it is bad idea to touch damaged FS before fsck.

Wait.

To run an unscheduled fsck of your root fs (and only your root fs), you have to remove the execute bit from a systemd component, kill that component, then remember to reset the bit after you're done?

That's nuts.

That kind of thing is why I stick with Slackware and OpenBSD for my production systems. systemd has some great ideas, but it has some down right silliness as well, and I personally don't feel it's stable enough for daily use yet.

(Just to make sure everyone gets it)

It will prevent it from running again. So it is basically a big FU! to systemd.

"So you you think you gonna restart yourself? Oh, no you won't"

That's nothing - systemd panics (pid1 crash) if you restart dbus, which requires either the Magic SysRq or the reset button..

https://forums.gentoo.org/viewtopic-p-7811550.html#7811550

Pottering was apparently told about this in 2013, when he simply declared it expected behavior because dbus was "rock solid".

http://lists.freedesktop.org/archives/systemd-devel/2013-Jan...

Incidentally, this might be the original reason for kdbus - it's harder to restart the daemon after it becomes a kernel module.

> Incidentally, this might be the original reason for kdbus...

Yeah, especially given that its stated reason (Moving into the kernel gives us a 2x speedup over userspace dbus! Dbus is slow and we NEEED the speedup!) is mooted by the 10x speedup that cleaning up the userspace dbus libraries has achieved. [0]

Having said that, in the systemd-devel post Poettering says that they're working on...

> ...improving this in two ways: even if it still brings down the system, at least allow logind to handle this case nicely, so that you get a sane getty. [The other way is to push dbus into the kernel.]

Systemd is a sprawling project, so I could see why it has taken more than two-and-a-half years to fix a local DOS caused by the perfectly normal method of responding to a DBUS upgrade. ;)

[0] Fun fact: Those cleaned up libraries were supposed to be released after kdbus was merged into the kernel in 4.1. The Systemd Cabal was so disappointed that they had to ship the performant userspace dbus libs before the dbus daemon got pushed into kernelspace.

I am by no means knowledgable in the relevant domain yet this is scary at so many levels. The team that is taking charge of so many critical infrastructure not doing a minimal level of fact finding before pushing for a new kernel module for RPC is a bit obscene.

I may have read things that are biased but I have a feeling it is a useful project and it does a lot of thing elegantly or right. But the project acts like it is the center of the universe and all should act to meet its needs. I have seen such project/modules at work, they usually makes everyone happy to start with and down the road when they finally get stuck and I am forced to consult we are already long way down the slope.

Goddamn, guys, stop repeating the same absurd.

kdbus is NOT ONLY about performance. It's about the lifetimes, the availiability in the early userspace, the new marshalling format, the new user bus concept, and so on, and so forth. It's about separating the transport (in the kernel) and the policy (in the userspace).

Most of these points can be achieved without putting the transport in the kernel, but some can not.

Yeah right. No wonder I see people adding all sorts of adjective to the systemd proponent. I don't usually go hunting for sources and I only had Linus's comment about kdbus and performance. So I went to look for pottering's own word and here is a verbatime quote from the abstract of his presentation on kdbus

  D-Bus is a powerful design. However, being mostly a userspace solution its latency and throughput are not ideal. A full transaction consisting of method call and method reply requires 10 (!) copy operations for the messages passed. It is only useful for transfer of control messages, and not capable of streaming larger amounts of data.

So yeah he tried to sell performance as the big thing for his version of dbus IPC. I'm guessing he (and group) spent more time with dbus than linus and failed to find the actual cause of the performance issue or tried to hide it give more credence to kdbus. In both cases it makes it that much harder to trust their reasoning and conclusions.

I never said kdbus is ONLY about performance. But from what I'm reading they are not performance people but they think they are. The most dangerous people are those who don't know their own limitation or weakness. And your comment simply emphasise the practice of listening whats being said. So yeah given that I don't know dbus or ipc I have to trust your logic and you only make it hard to do so.

Seems the quote gets turned into a single long line in Firefox, and subsequently making the page extra wide...

Strange. The quote is properly contained over here.

FF 40.0.3 on 32-bit Linux.

If you haven't, you should go back and read all of the LKML discussion surrounding the request to merge kdbus into Linux 4.1. Perhaps I misunderstood something, but I got the distinct impression that several of the folks reviewing the code were concerned that kdbus pushes too much policy into the kernel.

LWN article: https://lwn.net/Articles/641275/

LKML thread: http://thread.gmane.org/gmane.linux.kernel/1930358

> It's about the lifetimes, the availiability in the early userspace...

Starting dbus in your initrd [0] makes it available to the system from before the first second your service start machinery is started. If you want dbus available to userspace in your initrd, make starting dbus the first thing you do after you load your initrd. Is there some particular reason why this doesn't meet the lifetime and availability requirements?

[0] Remember that the initrd is responsible for things like decrypting the rootfs. When you're in the initrd, your real system is often not even remotely ready to run.

As mentioned above, that doesn't seem needed [ed: nm, it is - if journald is set to log somewhere other than /run that's usually mounted as a tmpfs].

[S]urely, something like:

  mount -t tmpfs /run && pkill journald

[ed: that is, for logs on /var/log: "mount -t tmpfs /var/log && pkill journald"] would make more sense? (In my experience systemd itself does something funky on / -- so the above wouldn't be enough -- but still seem a little more sane than running a chmod on a file (presumably) residing in an fs you want to fsck).

Actually, the workaround is to reboot with kernel param "emergency". It's single user mode with the root helpfully already in read-only mode.

The OS is Debian 8. I seemed to think the same too, but as I mentioned, rebooting into single user mode did no such thing.

You mentioned booting into rescue mode. Rescue mode is not emergency mode. Have a nearly two decades old explanation that still holds true for systems like nosh and systemd today:

> If you need to check the root file system, you can reboot into single-user mode with the root partition mounted read-only by issuing the -b switch at the LILO prompt. The -b switch will be passed through LILO to init and will cause an emergency boot ...

-- David A. Bandel (1997-01-01). Disk Maintenance under Linux. Linux Journal. http://linuxjournal.com/article/193

That points towards either Debian implementing systemd poorly in this aspect, Debian poorly documenting well how to accomplish this after the change, or a failure on the user's part to find said documentation.

Systemd on Fedora supports this perfectly fine, so I can't really call this a failure of systemd itself.

Maybe the problem is that you're carrying conventions from an obsolete system to a new one? A quick google search for how to force fsck with systemd comes up with results[1] mentioning options you haven't covered. Then again, I haven't had to do this, so maybe you did try those steps and it didn't work.

1: http://lmgtfy.com/?q=systemd+force+fsck

As far as I can see, they all mention "fsck.mode=force, as a kernel parameter", which is what I was complaining about.

Here's the TL;DR

1) It's actually super easy to get to a read-only root mode.

2) It actually is a problem of carrying over obsolete conventions from prior init systems.

3) It's not your fault because systemd has extremely poor documentation on this AFAICT, to the point that people searching for this problem can't even get answers from others that know how to do it, or at least those answers don't rate very high.

4) Just boot with kernel param "emergency" instead of "single" (or "rescue" which is an alias for "single") and you will get a single user mode with root already mounted read-only for you.

And now for the longer rambling version.

Okay, so I spent a little time on this since I happen to have a virt for Fedora 20, and I finally found a couple of solutions that work. First, the systemd solution, which is emergency mode. It's correct that the journal will cause problems here, but that's because the journal was already running and apparently systemd doesn't want to, or is unable to stop it. The solution is to reboot into emergency mode. This can be accomplished by using the kernel param "systemd.unit=emergency.target", or just the param "emergency". This will not only boot you into a single user mode, but will default to having root mounted read-only.

To clarify, this appears to be a new mode in addition to single user mode. Single user mode can still be reached with the kernel param "single" or "rescue" or "systemd.unit=rescue.target".

I finally stumbled on this when I decided to look at the directions for what I wanted to accomplish for the distro I was using[1] and not for systemd specifically. Which makes sense, but I think we are all to stuck on "systemd is different, how do I do this in systemd" rather than "how does my distro say I should accomplish this now, given the changes they've instituted."

1: https://docs.fedoraproject.org/en-US/Fedora/18/html/Installa...

2) It actually is a problem of carrying over obsolete conventions from prior init systems.

It actually is not. It's a problem of not knowing the conventions from prior init systems. emergency mode did not originate with systemd. It has been around since 1995-12-03 (Miquel van Smoorenburg's System 5 init clone, version 2.57d).

It was possible to boot to single user mode and remount root as read-only prior to systemd, Whether this was the optimal method for achieving this is irrelevant, it was possible, and popular. That specific method no longer works easily as before, while the optimal method does. The commentor was using their preferred (if suboptimal) convention from the prior init system and running into problems. I think that perfectly matches the statement "It actually is a problem of carrying over obsolete conventions from prior init systems."

It is also a case of not knowing the correct/optimal prior convention, but that doesn't make the original statement untrue.

This is a job for a distro like System Rescue CD. Boot off CD, run in DRAM, figure out which partition or LV is root, and directly fsck it.