Hacker News Reader

> It produces binary logs, which might not work with your workflow and is a kind of vendor lock-in if you don't script a binary-to-text script

I never really understood this point — it’s not some proprietary format that has to be reverse engineered, you quite literally has both encrypt and decrypt source code available for every single version it has been released. You can also just pipe the binary output through the provided “decrypt” tool and then further use the usual unix tools if you wish.

But let’s add the advantages of this logging: systemd can log events from the very start of the boot process, which was not possible before.

> But let’s add the advantages of this logging: systemd can log events from the very start of the boot process, which was not possible before.

From what I've seen using Alpine Linux for a few weeks on my Raspberry Pi, logs are written to /var/log/messages after the init process starts and launches the logging service. All logs before the init starts can be retrieved using dmesg? I'm not sure about this though, let me know if I'm wrong.

One of the things I haven't figured out yet is if traditional logging systems can easily do advanced log filtering like showing only logs from the current boot (like -b in systemd), previous boots (-b -1), and showing logs after a specific date and time (--since).

>One of the things I haven't figured out yet is if traditional logging systems can easily do advanced log filtering like showing only logs from the current boot (like -b in systemd), previous boots (-b -1), and showing logs after a specific date and time (--since).

I manage this with clever usage of grep. You are correct in that there isn't a single --flag that will only show me those specifics.

> I manage this with clever usage of grep. You are correct in that there isn't a single --flag that will only show me those specifics.

I can grep my way through text logs as well but being able to get logs for different purposes using --flags is better user experience. I can always resort to to using grep, sed, and awk if I want to when using journald but the loss of these quality of life features make it hard for me to consider using a distro that does not have systemd.

It’s even more difficult to understand when you consider than text files are themselves binary logs and given how many encoding they can have they are not even trivial ones.

People should just call systemd logs structured rather than binary because that’s what they are. Sure it means you need a shim if you want to view them as plain text but it’s not like we are talking about some unfathomable complexity here.

And you don't need journald to read the log files either, you can use `journalctl --file /path/to/file.journal` to read them directly via `journalctl`.

You can easily have it not produce binary logs.

Other software depending on Systemd isn't really Systemd responsibility. Its not systemd fault that almost nobody uses OpenRC.

I think its a bit of systemd's issue since it's attempting to cover a large range of functionality. The way it provides that functionality is important.

Re: the binary logs - true, but the core point that its not text by default is still a (small) issue IMO. Not ideal default behaviour.

There are solid technical reasons for systemd's binary logging.

The nice thing about its binary log format is that it's organised by fields which are indexed for quicker searching and filtering. It's much easier and faster to analyse these logs than the traditional text-based ones.

Also, having journald authenticate the process that is sending it log entries, and the log sealing capability, are two features that can help guard against log tampering.

> indexed for quicker searching and filtering

I think I am not the only one who is missing negative filters - check the log excluding audit messages.

Being able to boot a server with a 'live USB image'; mount root, and inspect the logs to see what happened are not possible when you got a binary blob for a log. When things have really fucked up, you need the 'ease of access that plain-text log files provides.

If your live USB image is from this decade, it will almost certainly come with journald for reading the logs. If it doesn't, why choose such a bad live USB image?

Just nitpicking, but AFAIK it's not journald which is used to read the logs, it's journalctl itself which directly reads the binary logs from the filesystem (and the normal filesystem ACL permissions are used to allow or deny reading the logs).

For me, the nice things about the systemd journal binary log format are:

- It's immediately compressed (without having to wait for a log rotation);

- It automatically rotates when a size limit is reached (no risk of filling the disk with logs);

- IIRC, it's deduplicated (repeated messages use less disk space).

All of these together means logs can be kept for much longer by default.

Overall agree. The only problem is if you have file corruption it could be a problem to parse. But I usually don't change the default.

Rsyslog can be configured with a database backend, if you want to index the log fields.

> You can easily have it not produce binary logs.

How do you do that? If you want non-binary logs, you can have journald tee log records to rsyslog; but the canonical log is still the binary log. Have I missed something?

What does canonical log even means here?

Systemd will produce logs in its own structured representation and write them straight away as plain text in the logger of your choice. For all intent and purpose, you now have text logs.

What I meant by "canonical" is that you can't get rid of the binary log; you can pipe it into a database, or a textfile, or /dev/null, if you want, but that database or textfile is transcoded copy of the binary log.

That's not true, you can configure journald to be in-memory only in which case there's no binary log, no file. If your using systems, you can't really git rid of journald, but you can get rid of the binary logs.

I wasn't aware of that. Can you make the memory usage arbitrarily small?

Every log ever is a transcodification of a system state. What difference does it actually make that it is emitted into systemd structured format before being put into your logger?

Seems like a pointless complain to me. It’s arguing for the sake of it.

Oh, I didn't complain. I know better than to come to HN and complain about systemd.

- It's also "big" in the sense that it's all-encompassing and hard to replace

- It makes alternating between systems harder because so much is Linux + Systemd specific.

> Its a "big" for small docker containers

Aside size, last I looked it required a lot of hoop-jumping to get systemd to run inside docker at all.

I think the idea of running systemd inside docker (or vice versa) is a little redundant to begin with—systemd and docker share a number of features.

It's unusual, but there are occasionally times when it's nice to run multiple processes in one container, and a proper service manager would be useful. Usually I see supervisord used for that kind of thing in practice.

Iirc using systemd in Podman is much easier.

nohup is POSIX, systemd is decidedly not. That makes it systemd’s responsibility not to break nohup, and more generally not to require non-admin users to be aware of it.

Not sure I fully understand this argument. Can you expand? I'm not familiar with nohup and its role.

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/n... blocks SIGHUP and captures output in a file. That lets me type

  nohup ./long_batch_job.sh &

and walk away, knowing that it will keep running after my shell exits and the controlling terminal is lost.

systemd broke that at some point, for no obvious reason. I think there is some nonstandard route to make systemd run something in the background, but end users shouldn’t need to talk to init.

Unix traditionally has never had proper seat and/or session management, with all the chaos this entails. With systemd, they added it. You’re effectively proposing that systemd (or anything else) should never implement session management. The most reasonable solution seems to me to be that nohup is instead adapted to do the semantically correct thing and tell systemd in the modern way that it should run the nohup command in a detached session.

I’m guessing you don’t mean setsid(2), but when you say “seat and/or session management,” what does that mean concretly? Why should init be involved instead of something under the user’s control?

How is systemd not under the user’s control? It is an init and service management system and long-running processes are services. Where else should it be managed? Systemd is the only component that knows whether you are logged in/out, whether you still have another seat open, etc. And its responsibility is to properly close resources, and not let unannounced processes to linger needlessly in the background.

init belongs to root. Only sysadmins can choose it (or not) and configure it, so only they should have reasons to interact with it or understand its UI.

systemd is usually a root process, regular user does not control it. Whether user being logged off means no such user processes should run is a policy question that has different answer on different systems and depends on the user as well. Systemd "unified solution" - killing user processes on logout - is not working well for many people especially on shared machines.

systemd-logind(1) manages user sessions. If you want to replicate the functionality of what “nohup” used to give you, I am reliably informed that “systemd-run --scope --user $command” does the right thing. Or, you can enable it permanently for your user with “loginctl enable-linger”.

When I see “manages user sessions,” I don’t know what that phrase means. What is a “session” (again assuming not a POSIX session leader) and what are we concretely doing to “manage” it?

Is there a guide that explains the bare minimum that typical end users need to know to stop systemd from breaking use cases that have worked for years? I would never expect a non-sysadmin to have read the “systemd-run” or “loginctl” manpages or know how to find them. I’m a nerd and even I don’t know how to get to the point of understanding what https://man7.org/linux/man-pages/man1/systemd-run.1.html is trying to tell me.

I mentioned systemd-logind. Its manual page systemd-logind(8) states¹: “See sd-login(3)² for information about the basic concepts of logind such as users, sessions and seats.”.

1. https://manpages.debian.org/stable/systemd/systemd-logind.8....

2. https://manpages.debian.org/bullseye/libsystemd-dev/sd-login...

> the bare minimum that typical end users need to know to stop systemd from breaking use cases that have worked for years?

That is bound to be a losing strategy. Systemd development (of breaking changes) is not stopping anytime soon. The simplest solution is to switch to an OS without systemd, like Devuan or Alpine or BSD.

> The most reasonable solution seems to me to be that nohup is instead adapted to do the semantically correct thing and tell systemd in the modern way that it should run the nohup command in a detached session.

That was proposed to nohup when systemd introduced the modification. It’s what nohup does on macOS.

A very productive discussion was happening but then the anti-systemd mob showed up with pitchforks, brigaded the discussion on the bug tracker and made sure nothing good would come out of it.

That’s when I personally decided Linux was a lost cause and definitely switched to macOS.

> I personally decided Linux was a lost cause and definitely switched to macOS.

That is an strange course of action. So because you saw some low quality discussion on the internet, you switched from free software you can influence to commercial one you can't?

I don’t care about having this kind of influence. These are tools. I want the tools I use to be good so I buy good tools.

It’s not about low quality discussion. A significant part of the Linux community seems to me to be deeply toxic and that leads to what I consider to be suboptimal technical choices.

> I want the tools I use to be good so I buy good tools

Right, good strategy.

> A significant part of the Linux community seems to me to be deeply toxic and that leads to what I consider to be suboptimal technical choices.

Do you mean toxic users influence some developers to make bad decisions? Any examples? Toxic users dumping on systemd may be one, but I don't think they have that much impact, systemd seems pretty successful in not giving away. Anyway systemd is a very specific case, this is not representative of all linux software projects.

They broke tmux too:

https://superuser.com/questions/1372963/how-do-i-keep-system...

Systemd does a lot of good but they like to move fast and break things. Making tmpdir per process, their homedir changes etc are all good for security but tend to break things in unexpected ways

They've never done tmpdir per process... There's a PrivateTmp option for services, but that has to be enabled explicitly

It is enabled by default on some distros.

Today you still have this functionality as an unprivileged user via systemd --user units in .config. Unlike nohup you get proper logging and processes can even restart automatically on reboot if you want.

Nohup is a disgusting hack from a previous century. I don’t see how having less information on what is happening to a process good — nohup interprets supposedly meaningful signals never meant for this misuse differently.

If you want to continue to run a process you have to start a service which is exactly that.

No, I don't have to do that at all, I can in fact use the standard unix way of making a process not terminate on SIGHUP by telling it to ignore SIGHUP.

SIGHUP is not a signal to shut down, it's a signal that says the controlling terminal is gone. The action taken by the process at that point is the decision of the process and the user that launched it, not the init system. This is exactly what it is for.

Changing that without understanding what people used it for across UNIX vendors for decades, was dumb. And it showed off just how little the writers of the service knew about how users actually use their systems. Huge red flag.

It's not just random long-running batch jobs it killed either, it seems that it would kill all sorts of useful stuff people run for reliability. The irony of the init system killing screen/tmux sessions which people run specifically to keep their shell going in case of a disconnect!

>And it showed off just how little the writers of the service knew about how users actually use their systems. Huge red flag.

Why do you assume they didn't know about this quite basic mechanism? They added 2 configuration options, one at compile time (to set the default) and another configurable by the user. In fact, distro maintainers didn't look up what it did and they just compiled it as is, resulting in the surprising behavior of killing lingering processes.

Also, it is an init and service management system and long-running processes are services. If you want to run something for "reliability" you should create a service for it, or just run the command with "systemd-run", to properly communicate with the responsible process manager what it supposed to do when the user logs out (and to actually make it reliable, e.g. you can then set it to auto-restart on failure, etc) Safely killing lingering processes and cleaning up after them is the correct default behavior, so systemd just tries to enforce that - they actually communicated with nohup maintainers to optionally add this systemd communication, so that "nohup whatever" would still work, but they were not open to it.

> distro maintainers didn't look up what it did

Even then, the default broke decades of standard behaviour so systemd failed users here.

> long-running processes are services.

Not all of them. A data processing/computation job running for tens or hundreds of hours is usually not a service. What is "long-running" anyway? Again, people used nohup for running their jobs for decades without being logged in. That is the standard usage of nohup, systemd broke it.

It might be the “correct” behaviour under systemd, it was not the correct behaviour under multiple other unix systems for decades, nor was it the expectation of Linux users or admins.

Paint it how you like, systemd broke decades-old, desirable behaviour for no good reason.

The attitude that everyone else is clearly wrong is unhelpful, and seems a red hat pathology. You see it with this and with Gnome, and it drives people away.