man journalctl journalctl -u nginx <G to skip to latest, ctrl+C because it's taking too long> journalctl | grep nginx <wait...> go back to man page to look for more options to search
etc
Really feels like "Embrace, extend, extinguish" but the thing they're extinguishing is unix-as-a-programming-environemnt.
Everything be in 4-digit or 5-digit error code.
Need to RTFM for such codes. Or worse, StackOverflow them.
A nice feature is that journald gives you flags like "--boot" to see logs only emitted during a specific boot, or "--since '5m ago'" which is not that straightforward to do with the approach you favor. (if there is a way to do it easily, please let me know!)
The difference is that you can use ls to find out, the same ls that you use when working with everything else, rather than needing to know some journald-specific thing that will only ever work for journald and will no doubt change again in another 5 years.
On my NAS
# strace journalctl -u nginx 2>&1 |grep open |wc -l
827
# strace journalctl -u baked-potato 2>&1 |grep open |wc -l
827
It's DB implementation is utter garbage. It doesn't even organize files by time or really anything useful. If the binary database was a SQLite file with columns containing app name/log level/etc. it would actually be useful but current one is just shit.
Bonus round:
#:/var/log/journal echo 3 >/proc/sys/vm/drop_caches
#:/var/log/journal time journalctl -u nginx >/dev/null 2>&1
real 0m3,232s
user 0m0,036s
sys 0m0,292s
#:/var/log/journal echo 3 >/proc/sys/vm/drop_caches
#:/var/log/journal time ag -R nginx . >/dev/null 2>&1
real 0m2,531s
user 0m0,004s
sys 0m0,290s
> It's DB implementation is utter garbage. It doesn't even organize files by time or really anything useful.
That sounds interesting, can you please elaborate on the internal structure of journald files or link to further documentation? And why would I care if journald handles it for me?
https://systemd.io/JOURNAL_FILE_FORMAT/
On the other hand I agree that journald seems to call open syscall more often than I would originally expect, which can be a problem for some edge cases[1], but I don't consider this to be a real problem.
If cache is hot, you have hundreds of MBs of systemd logs littering your buffer cache, while it could be used for something more useful
If cache is cold simple operations like systemctl status take up to tens of seconds, especially on loaded servers. Example:
# time systemctl status influxdb >/dev/null
real 0m11.913s
user 0m0.013s
sys 0m0.531s
Your example with dropping cache and running systemctl status is indeed interesting, 11s looks like too much. But it's a number without a context, and I wonder how big a problem this actually is. I haven't noticed it myself before.
> It doesn't even keep index of "last file where app wrote logs" which causes above.
While I definitely see some room for optimization, I'm not sure what you mean here: journald uses one active journald database file, there should not be a problem with figuring out where the file is, should it?
Well, I was routinely getting multi-second wait for operations like systemctl status servicename which is not something rare, because it didn't even kept the pointer to file that has last few lines of app's logs. It's not just benchmarks
It also trashes cache with hundreds of MBs of logs instead of stuff apps running on server actually needs. An equivalent of "tail -f" can go thru hundreds of megabytes of binary logs.
I do. I specifically want that as a more valuable and powerful feature. Undifferentiated, open-ended, general purpose, sharp tools are infinitely more useful.
It's a big difference, not because one command, but because tail is part of the standard unix toolset that works with all files. Except now you have this one log file that is special and needs a different unique tool.
> ctrl+C because it's taking too long
Try --since=today. Then you end up in less, and that should be more familiar, including for search. This is then the equivalent of traditional daily rotating log files, with no other knowledge needed apart from --since and --unit (the latter is noted in "systemctl status" output as a reminder).
Then every further feature is an optional bonus.
It's the same rule I use when purchasing kitchen equipment - often it's better to have a single multi-purpose tool than a bunch of specific implements. Otherwise you end up with a drawer full of strawberry-stem removers, pizza slicers, corn-cob peelers, avocado-slicers, etc.
This is to say nothing of the binary format issues that others have discussed.
The backing store may be controversial, but the journalctl command line is much more pleasant. "I want to see log messages from the end of the last boot when the system crashed" is no longer a Rube Goldberg command line involving less/sed/grep/cut/awk and manual scrolling, it's supported out of the box with no configuration necessary.