Hacker News Reader

Comment by AshamedCaptain | Hacker News Reader

I have a system where systemd decides, about half the times I boot it, to immediately unmount all of the filesystems listed in fstab literally right after having finished mounted them as part of the regular boot process. All of this with other daemons starting in parallel.

It will fail to unmount most of the filesystems, since they are of course busy, but often it will succeed in unmounting /var, /tmp, /home and others. Then it will continue starting on further services as if nothing happened and even proceed to gdm. But of course without /home I can't even login.

No idea how to even start debugging it.

View on HN

That's super annoying. I've experienced similarly frustrating boot problems when I had some FS corruption causing /var fsck to fail @ boot.

Despite dropping me to a rescue shell, systemd entered a continuous loop of retrying fscking /var while I was attempting to identify and fsck the problematic filesystem.

Few things are more infuriating than having your system drop you to a rescue shell, while it continues to endlessly change the state of the system - and not just any state, but the very area you're in the rescue shell to investigate and resolve. It just kept making /var busy while I kept attempting to manually fsck it...

I would run systemd-analyze critical-chain and then systemd-analyze blame to see if there is a service that could do this. And then I would look at the detailed boot logs. If nothing can be found, I'd look for some way to increase the log level.

If you are not curious, reinstalling everything could be quite fast too, depending on what you have installed.

The logs are mostly useless and random. There is nothing that consistently appears between the point where it finishes mounting (lots of "Mouting...", then "Reached target Local File Systems") and where it starts unmounting ("Stopped target Local File Systems", then lots of "Unmouting ..."). Sometimes these two lines are less than one second apart. But I haven't found any message which is consistently between these two points that would point to a suspect.

And this is not a problem that happens on every boot, so it doesn't show up on analysis... not to mention that most of the analysis is designed on trying to reduce boottime, which obviously is no help (thought it is rather fast already).

It is rather easy to guess why a service is starting, but not why it decides to stop 'orderly', much less why a .target decides to stop...

Most certainly you can only analyze things on boots where the problem appears, unless logs are persisted across boots.

Indeed systemd-analyze focuses on boot times, though I find it is useful to get a list of services that are run.

If you are adventurous you could probably replace the umount (resp. the systemd-umount) binary by a script that runs umount (resp. systemd-umount), and also prints the process tree (and its PID) in some file so you can get insight on what ran it.

You can check in journal which service startup does the unmounting align with. Maybe generate a bootchart http://manpages.ubuntu.com/manpages/bionic/man1/systemd-boot...

You could also replace umount with a wrapper that logs the parent process tree on execution.

And of course you can open an issue on GitHub.

> And of course you can open an issue on GitHub.

Only do this if you've reproduced what seems to be a bug in one of the latest two releases, otherwise contact your distro.

The systemd github issue tracker is not a support forum, you'll find this stated in the issue submission form.

Fair advice, and they’re probably right to redirect most users that would file a bug with them.

But on a normative level, perhaps a project that doesn’t want to support users shouldn’t have subsumed a huge swath of user land things?

> But on a normative level, perhaps a project that doesn’t want to support users shouldn’t have subsumed a huge swath of user land things?

It's not that the project doesn't want to support users in some absolute sense, it just doesn't want it happening in github issues.

The project provides the systemd-devel mailing list for that, but it's still preferred that users stuck on old versions determined by their distro's release cadence seek support from their distro vendor. It's likely such slow-moving distros are patching their LTS versions anyways, and are best positioned to support what they ship.

If this is a bug in systemd it belongs at systemd no matter which version it is. If systemd doesn't want to do support that is their choice but distros aren't the right place to post systemd bugs. It only belongs there if it is either a bug in the distro itself or a support request for distro specific problems.

If you go filing bugs against old versions of systemd in the github issues, you're likely just going to receive a canned reply along the same lines of what I've already said.

You may not like it, I'm just playing messenger here, to try save some bother on both sides of this subject. Use the mailing list for support questions, or contact your distro vendor for support.

We had that kind of problem because we relied on /dev/disks/by-label

Can you tell us more about this? How did you determine that that was the problem? What was your solution?

I ask because I use /dev/disks/by-label and have never had this problem, and if I ever had to experience it, it would drive me insane.

I've solved a similarly sounding issue by using the UUID. Get it with blkid, then use UUID="<uuid goes here>" in fstab instead of LABEL="<label>".

But this was a long time ago, on a much older systemd (like Fedora 17 or 18).

This is a common error even back in the Sysvinit days. /dev/disks/by-label is not guaranteed to map the name labels to disks on every boot, and it can change with boot timing based on the storage controller behavior. Use /dev/disks/by-uuid instead.

/dev/disks/by-label is a legacy from the old days of systems with static disks wired in a specific, unchanging configuration.

You've confused by-label with the kernel's device nodes (e.g. /dev/sda1). Kernel names are assigned in order. Labels are just like UUIDs and part of the filesystem.

However, unlike UUIDs they don't necessarily exist, are not as likely to be unique (you should only get duplicate UUIDs by cloning filesystems) and are easier to change. This makes them less suitable for mount configuration.

Long time ago now, but we had disks just unmounting and mounting at free will.

As someone said by-label is not stable and could be changed, causing problems. We still use it though but not in customer facing servers where these problems arise. Maybe related to lvm/snapshot etc.

Deactivate graphical boot screens (often just by pressing Esc), increase the log level[0] and try to read the logs somehow. Maybe you can still log in as root. If you cannot log in as root even, but networking works, configure rsyslog to send your logs to another host and read them there.

[0] https://freedesktop.org/wiki/Software/systemd/Debugging/

if you can't login as root, you can try to put bash as init (with init=/bin/bash in the kernel boot command line) to inspect the system. Might be difficult on some systems (with SELinux?)

if you have access to kernel command line to add init= you can also disable selinux there.

I'd rather recommend booting a liveCD. grml, knoppix, whatever floats your boat. An init=/bin/bash shell needs quite a bit of knowledge to get working (mount rw, get rid of selinux, mount the real rootfs), is possibly just busybox and generally a pain. Even more so if you need the net to copy something over.

It isn't impossible, and OK in a pinch, but why make it hard on yourself?

You can actually specify systemd.unit=rescue.target nowadays.

https://ostechnix.com/how-to-boot-into-rescue-mode-or-emerge...

You forgot mounting /proc and /sys :) Anyway, it's the quickest way to recover if all you need is to edit some config file or run passwd.

Have you tried increasing the log-level in /etc/systemd/system.conf? You can also set LogTarget=kmsg if you have issues with the journal. You can also control this with the kernel command line if you prefer https://www.freedesktop.org/software/systemd/man/systemd.htm... You could try a kernel command line like this 'systemd.log_level=debug systemd.log_target=kmsg' for instance

You can also look at systemctl list-dependencies local-fs.target to see if it has any failed dependencies

You can also use systemctl show local-fs.target to sanity-check it to see if there are any local modifications to the target that are breaking it

I'd start by looking at the systemd unit files in /etc/systemd/system. Beware of service wants.

I had straight up errors in a network unit file that caused weird behavior (IP address disappearing). This was a long time ago though. Sometimes moving a unit from one target (graphical) to an earlier (multiuser) solves the issue.

You might also have "defaults" in /usr/lib/systemd/ to compare with.

systemctl and friends are really good tools for troubleshooting but takes a little while to get used to. Duck duck go is your friend.

Finally, some volume technologies like VDO requires a x-systemd.requires statement in fstab.

Systemd generates mount units for the filesystems listed in fstab. Maybe check those with systemctl status? (Like `systemctl status -- -.mount` for my root partition)

Candidly: that command looks like the least discoverable command I’ve seen in the last year.

The command

    systemctl status *.mount

Just shows the status of all mount units.

It's the same command you'd use to check any other systemd unit, for example, if you want to check Docker's status:

   systemctl status docker.service

Seems consistent and discoverable as long as you have a surface understanding of systemd units.

That to me reads as "show me the status of all files ending in .mount in the current directory". Once you start hijacking shell metacharacters you're on dodgy ground.

It's not using the shell glob syntax, it's just a regex compatible string. The find utility does the same thing:

    find ~/Code -type f -name *.js

Where *.js isn't using the shell glob feature, it's just a regex compatible string.

You could put quotes around it to make it clearer:

    systemctl status "*.mount"

The string is matching on unit names, and the mount units have the .mount suffix.

> Where *.js isn't using the shell glob feature, it's just a regex compatible string.

Except it isn't, because a `*` at the start of a regex is invalid. `-name` patterns given to `find` are explicitly shell glob patterns. You might be thinking of `-regex` patterns, which GNU find also has.

It's a bit shonky when `find` does it because it's not saying "the shell pattern applies in this directory", and I regularly trip over it. It's something special you've got to know about how `find` works, but at least it's still anchored to the idea of being searched in a path that's also part of the command. There's a semantic link there.

With `systemctl status *.mount` you've not even got that. What path is being searched for `*.mount` files? Is that path part of the `systemctl` interface? Or is this just another way that `systemctl` insists on being a special snowflake that someone's decided I now need to devote neurons to?

Fair. I think it's because systemd lets you use slashes in unit names, but you can't represent those as files so you have to use a dash instead. This is the worst for the root filesystem mounted at / since that's the entire name. In systemd's defense, it warns you when you try systemctl status the normal way.

  Hint: to specify units starting with a dash, use "--":
        systemctl [OPTIONS...] COMMAND -- -.mount ...

I found my mount names by running systemctl list-units, so it's not something I had to look up.

You can use systemd-escape(1) to have it generate the correct unit name for you, e.g.: systemctl status $(systemd-escape --path /home --suffix mount) Yeah, complicated and verbose, but reliable and integrates into systemd mindset very well.

It is at least built in a logical way and once you understand that logic then related commands will be discoverable.

`systemctl` is the systemd command line tool for inspecting and manipulating services/units. `systemctl status` is the command for showing the status of a service or unit.

The `--` part is not specific to systemd at all, it's used in many commands to separate flags from positional arguments. This lets you use positional arguments that might happen to start with a dash without them being interpreted as flags. [1].

systemd manages mounts, they're named according to the mount point, with a transformation to turn it into a filename (mostly converting slashes to dashes, plus a .mount suffix). Hence, the root mount ('/') is named '-.mount'.

[1] See, e.g., https://unix.stackexchange.com/questions/11376/what-does-dou...

The annoying part is that now that '-' stands for '/', you need to use '\x2d' for '-'. Which is pretty annoying when manipulating mount names during the shell.

I wish they had picked ':' for '/' instead of '-'. That one would be pretty uncommon to find in a mount path, and has a bit of precedent on OSX.

(Who would ever mount anything on a path with a colon anyway? That would be as silly as using slashes instead of dashes for options)

C:\ much?

I’m pretty sure whoever came up with that was also using slashes for options. Someone was gonna bite :)

Have you seen system-escape?

https://www.freedesktop.org/software/systemd/man/systemd-esc...

Would

  systemctl status -- $(systemd-escape --suffix=mount --path /)

be simpler to read?

If you use a wildcard operator, you can list all the mount statuses: systemctl status *.mount

Sadly no. The problem is that the entire local-fs.target is stopped 'cleanly' but with no clear reason. i.e.

    $ systemctl status local-fs.target
    Active: inactive (dead) since [...]
    21:12:56 ... systemd[1]: Reached target Local File Systems.
    21:12:57 ... systemd[1]: Stopped target Local File Systems.

Yes, the log just shows it stopping one second after being started, nothing happening inbetween, and no reason given for it being stopped.

Incredibly enough, local-fs.target is a dependency for graphical.target, so the system should not have continued booting. But not only it has continued booting, systemd even thinks that it finished booting all-OK. State is "running" (not "degraded" as it would be if any service/mount failed), with 0 failed services. Even though both graphical.target and local-fs.target are 'dead'.

Complex systems fail in complex ways.

If that happens again I'd try opening a virtual terminal, login as root and check logs.

> If that happens again I'd try opening a virtual terminal, login as root and check logs.

If the system has unmounted all the drives, there won't be any logs. This happens often enough with systemd / journald to be a big source of annoyance.

The only option you get is to tell systemd to boot in debug mode but that fundamentally alters the timing and behaviour of so much that it often stops the weird behaviour from ever happening (similar fun can be had with `dracut` where enabling debug tends to stop any race condition from ever happening)

Note systemd does dump its own log to the kernel ring buffer until journald starts (the last message I see on the kernel ring buffer is Started Journal Service). It's no help though when (as in my case) journald starts but then has /var unmounted below its feet. I can write a script that keeps /var busy though, preventing it from being unmounted.

> I can write a script

There's some delicious irony in having to write a shell script to combat the init system that was designed to replace all the shell scripts.

Reply on news.ycombinator.com