Comment by 0x_rs | Hacker News Reader

Comment by 0x_rs | Hacker News Reader

Parent Full thread

journald is awful for many reasons, but what makes it worse is that everything running on your machine thinks it has any rights to dump all the logs it wants unprompted. Open a file picker and kio will decide it's a good idea to spam tens or hundreds of thousands of entries into it a day, listing every single file you have in a directory with some log such as "No node found for item that was just removed" and that has zero impact to the user whatsoever. You almost need to keep a script tracking all the journal floods for every new service to make sure it's not treating your system log as its dumping ground. To be fair, the kernel and usb peripherals can also have a bad day and spam 3 million lines an hour into it, think input irq status -75.

It's too much of a chore to keep up with all the program-level configs (if they have them) and service files, but LogFilterPatterns in systemd can help in an unintended way: you can make one log blacklist with a .conf file in /etc/systemd/system/service.d/, and put in there all the patterns that spam your journal one by one, don't even have to chase misattributed loglevels. It just looks something like:

[Service]

LogFilterPatterns=~I am a completely useless log entry

LogFilterPatterns=~I am another useless log entry

But it doesn't pick up on identifiers and doesn't do anything for kernel spam. It's only great to make some messages shut up. Also, I'd consider any btrfs install that does not have nocow on cache, journal etc. to be defective.

That was the task for years for syslog services that dealt with it without issue.

I can confirm this, used it for many years, 3 keywords: efficient, reliable, useful.

all 3 missing on journald,in my experience i saw it inefficient also on configuration level, unreliable because of loosing loglines on crash or reboot and not useful since to look at logs i need 3 commands, verbose parameters and 5 google search to find them.

syslog experience? very efficient also on heavy load production instances, never lost a log, pipe grep and jq and you have the info you need.

so what I experienced is that a default linux install was shipping a rock solid logging system by default, reliable and usable and everybody knew what was where and you will find it. now i just have fancy stuff, units etc and lost all of that.

no I dont need to tune config parameters on a default install to have working basic logging tnx.

> syslog experience? very efficient also on heavy load production instances, never lost a log,

You’re lucky. The original syslog protocol was fire-and-forget UDP (which I believe is still the default, though it’s been ages and I could be wrong) and the daemon was single threaded. I/O or CPU starvation could easily lead to dropped logs.

WHen I provisioned a mid-range dedicated box recently, I went FreeBSD almost entirely so as to not have to touch or deal with anything related to systemd, the worst the thing to ever happen to linux.

The problem with systemd is the scale. Some of it is fine, some of it is actually quite good. But other areas are just replacing existing systems with things worse for the majority of traditional users. Logging, time and dns come to mind.

systemd-boot ia actually one of the rare components that are pleasant to work with.

Gentoo also has a systemd-free option (and even if it didn't, you could make one because Gentoo is basically LinuxFromScratch)

WHAT R UR CFLAGS??!?

https://www.devuan.org/

And still does. I generally replace journald with rsyslogd on systemd setups.

The syslog architecture never had a filter component in the middle that could drop logs before they reached syslogd.

rsyslog is an incredible piece of software. Every time I'm looking for something to do with logs, opening the docs or googling finds the feature for me and myriads alternatives. I know it still exists and use it heavily on any system I'm in charge of, but there's some regret at having a dual system with journalctl...

syslog wasn't dealing with it because all of this output was written in .xsession-errors in user's homedir (which was not rotated btw)

> I'd consider any btrfs install that does not have nocow on...to be defective.

You're getting COW on the extents if you're snapshotting anyway.

systemd-journald also has rate limits that you can configure ;-)

IIRC those are "dumb" rate limits though, aren't they? I.e. if 1000 of the same dumb message comes across from the service then you'll rate limit out the 3 useful messages occurring at the same time.

I'm using a certain object storage implementation post minio enshitification. The software itself is great don't get me wrong but I've noticed their logs are basically unreadable. Its metrics and traces in json data meant to be rendered on a dashboard instead of being read by humans. It's also extremely verbose even at an INFO level.

Maybe just get these on an open telemetry endpoint instead? I also don't get why people send by default json logs to journald as it's clearly meant to be a replacement to syslog which is already a good standard.

Reply on news.ycombinator.com