Systemd-journald: possible risks and limitations

Debian's decision to use systemd-journald as the sole and default log collector starting with Debian Bookworm has sparked some concerns among system administrators. While systemd has been widely adopted due to its comprehensive approach to modern problems, the use of it may cause unforeseen problems for inexperienced administrators.

remark

In the following, I want to address some possible risks and limitations. I'm aware that this text represets my opinions and concerns. Different system administrators or users may have varying perspectives on systemd-journald and its suitability for their specific needs and I would love to discuss with you.

issues

In my opinion, systemd-journald has three issues:

1) The binary format lacks self-healing features and the provided tools can't deal with corrupted logs.

2) There are currently no public forensic tools to recover corrupted binaries.

3) It is impossible to delete specific units or redact data sets.

Regarding 1)

In case of a journald log corruption due to a crash or a bitflip, the provided tool journalctl can't read or repair the binary log, nor does it even warn of detected corruption by default (even switch `--verify` doesn't do well). I brought this to systemd's attention back in 2021 [1] (the bug dates back to 2014 [2]) and Lennart Poettering responded himself / closed the issue himself. Funny side note: After trying to discuss his reply in the IRC channel "systemd" on Libera, I got permanently banned.

Regarding 2)

I'm unaware of tools that can recover corrupted binaries. Existing parsers likely rely on systemd's API, which has the same limitations. Although systend-journald has been around for 10 years, the binary format has been changed several times, but this seems to have found no takers outside the core project.

Regarding 3)

For me the most serious limitation. It isn't possible to delete specific units or edit entries. You can't selectively delete telemetry of a dev-container, nor redact personal data to comply with GDPR. The options are to delete all logs or to suffer noncompliance. In terms of GDPR, you have to wipe ALL your logs just because one person asked to delete his data. Another example: You also can't remove the access logs after 2 days and keep the error logs for 10 more days. The systemd community suggests converting binary logs to line-delimited text, editing via text tools (sed), then converting back. However, logs continue accumulating, making this infeasible in practice. This procedure isn't documented and lacks safeguards to prevent data loss or new corruption.

conclusion

While systemd has its advantages, it's important to carefully consider the potential risks and limitations. I would recommend sticking with rsyslog, syslogd or another log collector instead of relying solely on default "systemd-journald only".

references

[1] https://github.com/systemd/systemd/issues/19875

[2] https://bugzilla.redhat.com/show_bug.cgi?id=115735