Hacker News Reader

Comment by fullstop | Hacker News Reader

I was forced to use systemd and journald for a while and I've seriously grown to like them, especially journald.

They’re really nice until everything on the node is completely broken. Then they are a massive obstruction to access and understanding what went wrong due to the opaqueness. It’s best to do some drills on a purposely broken system to gain some deep insight into recovery scenarios.

If everything is broken and you need to do recovery, don't you usually do that from another system anywhere? And mount the target system.

I imagine the journald logs are just files at the end of the day and you can just read them with some tool?

This can be tricky because journald compression, but is this really that much different than a corrupted compressed log file, like that which logrotate produces, in /var/log?

If that is your concern, you can disable compression in the journald configuration so that the contents can be read with "strings" or similar tools.

I am curious what kind of problems have you seen? I am transitioning some of my systems to journald, and I am very interested in things that could go wrong.

So far, I tried looking at logs from the dead system using “journalctl -D” - it seemed to work. And the way how the log files from each boot are always separate is pretty handy. Other than that, the only problems I have seen were having more to type and having to learn more commands.

Am I in for a nasty surprise?

Key issues for me were machine-id related and journal corruption. Also journalctl dumping core after a recovery-mode boot. That left us with no tools to deal with a compressed journal and no strace to find out what it was doing. I actually attached a USB disk to the machine and cp’ed the files off it in the end. This was inconvenient as the node was 1550 miles away from me.

Did you use -D / --directory option to journalctl? It switches the tool from "do the common thing and inspect live system" mode to "do exactly what I want and just convert files in that directory to text". There is also "--file" option for one-file-at a time access, but it is harder to use.

In particular, I have just verified with "strace" that running journalctl with either "-D /var/log/journal" or "--file" option does not even open machine-id file nor a d-bus socket -- so whatever problems you had with machine-id would be gone with.

Also, when you said, that you "cp'ed the files off it", did you mean you copied off the contents of /var/log/journal? Were you able to open the files on the other machine?

If your logs are that important they should get shipped off the node immediately anyway. That's what ELK, Graylog, loki etc are for. If you don't do centralized automated log management on a many-node server farm and that impedes your forensic processes when a node fails then not compressed logs are to blame but your system. On any server I am responsible for, any log that is created on the node and required or useful for forensics gets shipped off the node immediately after it's written, including all system logs, container outputs, etc. I can only highly recommend it. The amounts of data to store centrally are very manageable as well even if you ship 100% of logs.

So, if you care about logs, don't depend on journalctl... Well that's fine if you can ship logs off the box, and if the log shipping is working. But if you have to troubleshoot on the machine itself (hello, desktops?) then having tools that don't come with a lot of pain and big learning curve is critical to addressing the issue quickly. What's the point in even having these tools if we're not supposed to use them?

Normally, Unix-like tools are not as painful as journalctl. But Poettering's interfaces are a gauntlet of unusual concepts and hidden inter-relationships, with no common examples or intuitiveness. And things like dbus make it worse, now that there are many parts of a modern Linux system that have no console interface, because nobody's written one for the particular application you need to view or change the right dbus settings. And /sys/ is literally a wilderness of random undocumented settings that are often the only interface to critical system functions.

Linux distributions are now a tiresome no-mans-land of overcomplicated mysterious crap. I'm willing to bet the major Linux distributions will be abandoned over the next decade for simpler systems that are cloud-native, mobile-friendly, and have less Kafkaesque interfaces.

What kind of desktop do you have where your journal is constantly getting corrupted? Most distributions log both to the journal and to /var/log/, so I'm seriously struggling to understand the complaint here. GP's message about centralized logging is saying that if logs are that important to you, you will keep them stored in a centralized location. You should be able to catch the server on fire and have logs recorded centrally up until that point. This would be the case with or without journald.

You started with a strawman about caring about logs, and ended complaining about sysfs.

Yeah I digressed quite a bit, sorry.

> I'm seriously struggling to understand the complaint here

The complaint was trying to explain a parent commenter's point about journald "They’re really nice until everything on the node is completely broken. Then they are a massive obstruction to access and understanding what went wrong due to the opaqueness."

Point: Logs are really annoying to manage on systems with journald.

Counter-point: Ship your logs somewhere else / you probably don't have these problems in real life

Counter-Counter: If we're not supposed to use these tools on our hosts, exactly why are they installed?

Logs are written to both the journal and /var/log/

I would argue that logs are less annoying to manage on systems with journald, once you take the time to learn how to leverage the tools.

I would also argue that shipping mission-critical logs off-server is a worthy endeavor, regardless of logging system used.

I like journald because it lets me isolate logs for a particular unit without grepping and accidentally including output from unrelated services. It's faster to find the data that I need between time ranges rather than manually comparing time stamps.

I can count on one finger the number of times the journal has been corrupted on the servers that I manage, and it was because of hardware failure.

I have this fantasy that all the pro systemd posts are really just bots run by leonard

I, Leonard, do not know of what you speak. ;-)

Yes 100% agree with this and we do that but there’s normally a few seconds window between something going snap and being flushed to the upstream log aggregator. At which point the smouldering remains of your node or container are fairly important to have at hand.

It’s like having a black box that forgets the last 30 seconds of the flight otherwise.

This is incredibly difficult to get right.

Which has better odds, the smoldering remains having actually flushed the write to disk or the flushed to the upstream log aggregator?

From experience of both, the smouldering remains seems to win out by a fair margin. The aggregator may have some smoke signals in it but that’s about it.

Most of the issues I’ve dealt with shaft the network before the filesystem.

Outside of the "cloud world" most of my failures have been the result of disk failure. In those cases, the logs definitely did not get written to disk, but they were flushed to graylog.

I can't remember the name of it anymore, but we used to use a user-space daemon to collect logs from applications over various methods (sockets pipes files etc) and it would store them in a ring buffer until they could be flushed to disk or network. If neither happened, you could attach to the host somehow and get the logs out of memory, watch them on a terminal screen, etc.

Opening an EC2 console in AWS is still a simple and reliable way to find out what's going wrong with your instance. Wouldn't be possible if we didn't already have the convention to have the kernel, syslog, etc print to tty1.

Yep

Reply on news.ycombinator.com