No really I don't think journactl makes sense in its current form. It's just broken by design.
I do like the potential of it. But not the implementation.
> journalctl --grep is still much slower than grep on simple files
My results don't match your conclusion.
> if you use ripgrep
The maintainer also has been pretty rude to me about this on HN. I can get we have a different opinion but it's still crazy to think people won't be caught off guard by this behavior. Its name is literally indicating it's a grep replacement. Yeah, I'm surprised its behavior significantly diverges from grep lol
This is AFAIK the only other interaction we've had: https://news.ycombinator.com/item?id=41051587
If there are other interactions we've had, feel free to link them. Then others can decide how rude I'm being instead of relying only on your characterization.
> but it's still crazy to think people won't be caught off guard by this behavior
Straw-manning is also crazy. :-) People have and will absolutely be caught off guard by the behavior. On the flip side, as I said 9 months ago, ripgrep's default behavior is easily one of the most cited positive features of ripgrep aside from its performance.
The other crazy thing here is... you don't have to use ripgrep! It is very specifically intended as a departure from traditional grep behavior. Because if you want traditional grep behavior, then you can just use grep. Hence why ripgrep's binary name is not `grep`, unlike the many implementations of POSIX grep.
> Its name is literally indicating it's a grep replacement.
I also tried to correct this 9 months ago too. See also: https://github.com/BurntSushi/ripgrep/blob/master/FAQ.md#pos...
For anyone else following along at home, if you want ripgrep to search the same files that GNU grep searches, then do `rg -uuu`. Or, if you don't want ripgrep to respect your gitignores but ignore hidden and binary files, then do `rg -u`.
It makes sense that folks might be caught off guard by ripgrep's default filtering. This is why I try to mitigate it by stating very clearly that it is going to ignore stuff by default in the first one or two sentences about ripgrep (README, man page, CHANGELOG, project description). I also try to mitigate it by making it very easy to disable this default behavior. These mitigations exist precisely because I know the default behavior can be surprising, in direct contradiction to "but it's still crazy to think people won't be caught off guard by this behavior."
Forgive me if I'm a bit surprised!
I still stand that silent errors are significantly worse than loud ones
| it's worse to not get files you're expecting vs get more files than you're expecting. In the later case there's a pretty clear indication you need to filter while in the former there's no signal that anything is wrong. This is objectively a worse case.
> The other crazy thing here is... you don't have to use ripgrep!
I don't think grep ignoring .gitignore files is "a bug". Like you said, defaults matter. Like I said, build artifacts are one of the most common things for me to grep.
Where we strongly disagree is that I believe aliases should be used to add functionality, where you believe that it should be used to remove functionality. I don't want to start another fight (so not linking the last). We're never going to see eye-to-eye on this issue so there's no reason to rehash it.
I don't either? Like... wat. Lol.
> Where we strongly disagree is that I believe aliases should be used to add functionality, where you believe that it should be used to remove functionality.
Not universally, not at all! There's plenty of other stuff in ripgrep that you need to opt into that isn't enabled by default (like trimming long lines). There's also counter examples in GNU grep itself. For example, you have to opt out of GNU grep's default mode of replacing NUL bytes with newline terminators via the `-a/--text` flag (which is not part of POSIX).
Instead what I try to do is look at the pros and cons of specific behaviors on their own. I'm also willing to take risks. We already have lots of standard grep tools to choose from. ripgrep takes a different approach and tons of users appreciate that behavior.
> We're never going to see eye-to-eye on this issue so there's no reason to rehash it.
Oh I'm happy not to rehash it. But I will defend my name and seek to clarify claims about stuff I've built. So if you don't want to rehash it, then don't. I won't seek you specifically out.
> I don't want to start another fight (so not linking the last).
To be clear, I would link it if I knew what you were referring to. I linked our other interaction by doing a web search for `site:news.ycombinator.com "burntsushi" "godelski"`.
> If it wasn't clear, I don't ;)
OK, so you don't use ripgrep. But you're complaining about it on a public forum. Calling me rude. Calling me creepy. And then whinging about not wanting to rehash things. I mean c'mon buddy. Totally cool to complain even if you don't use it, but don't get all shocked pikachu when I chime in to clarify things you've said.
> Calling me creepy
I said it was creepy that you appears seemingly out of nowhere is a very unexpected place.
I'm only giving this distinction because this category of error has happened a few times.
This is just rhetorics.
Yes, it is creepy when someone randomly appears just after you allude to them. It is also creepy when someone appears out of nowhere to make their same point. Neither of you were participating in this thread and appeared deep in a conversation. Yeah, that sure seems like unlikely circumstances to me and thus creepy.
Yeah, but storing logs in binary and having a specific tool to just read them is sure not a crazy design choice.
Alright. Let me entertain you.
In the data I provided, counting the lines in a big log file was 469.5 times faster than journalctl took to output all the logs.
From this information alone, it seems difficult to believe that journalctl --grep can be faster. Both had to read every single line of logs.
But it was on a rather slow machine, and a couple years ago.
Here /var/log and the current directory are on a "Samsung SSD 960 PRO 512GB" plugged via m2 nvme, formatted in ext4 and only 5% used. Though this shouldn't matter as I ran every command twice and collected the second run. To ensure fairness with everything in cache. The machine had 26GiB of buffer/cache in RAM during the test, indicating that everything is coming from the cache.
In my tests, journalctl was ~107 times slower than rg and ~21 times slower than grep: - journalctl: 10.631s - grep: 0.505s - rg: 0.099s
journactl also requires 4GiB of storage to store 605MB of logs. I suppose there is an inefficient key/value for every log line or something.
For some reason journalctl also returned only 273 out of 25402 lines. It only returns one type of message "session closed/opened" but not the rest. Even though it gave me all the logs in the first place without `--grep`?!
Let me know if I am still using it wrong.
$ sudo hdparm -tT /dev/nvme0n1
/dev/nvme0n1:
Timing cached reads: 33022 MB in 1.99 seconds = 16612.96 MB/sec
Timing buffered disk reads: 2342 MB in 3.00 seconds = 780.37 MB/sec
$ du -hsc /var/log/journal
4.0G /var/log/journal
4.0G total
$ time journalctl > logs
real 0m31.429s
user 0m28.739s
sys 0m1.581s
$ du -h logs
605M logs
$ time wc -l logs
3932131 logs
real 0m0.146s
user 0m0.065s
sys 0m0.073s
$ time journalctl --grep=sshd | wc -l
273
real 0m10.631s
user 0m10.460s
sys 0m0.172s
$ time rg sshd logs | wc -l
25402
real 0m0.099s
user 0m0.042s
sys 0m0.059s
$ time grep sshd logs | wc -l
25402
real 0m0.505s
user 0m0.425s
sys 0m0.085s
> Let me know if I am still using it wrong.
Check out the filetype of the journal files
$ file -bi /var/log/journal/1234blahblah/system@foobardedoopdedo.journal
application/x-linux-journal; charset=binary
> it seems difficult to believe that journalctl --grep can be faster.
P.S.: I know. As I specified in my earlier comment, I get burned with build artifacts and project logs. Things that most people would have in their .gitignore files but you can sure expect to grep through when debugging.
THEY ARE NOT DOING THE SAME THING
> that if the logs were just stored in plain text
cat /etc/systemd/journald.conf
https://www.freedesktop.org/software/systemd/man/latest/journald.conf.htmlFor the exact same logs:
journalctl takes 10s to search through 4GiB. And misses most of the matches.
(rip)grep takes (0.1)0.5s to search through 605MiB.
In other words. journalctl consumes much more space, and a lot more time, to return an order or magnitude less results to the same query.
What does journalctl offers, that makes this resource, time and correctness tradeoff worth it?
The results in your comment aren't measuring the same thing. There's no grep on the /tmp/all.log in the middle code block, which is the thing they're talking about comparing.
Btw, you can choose not to store journald files as compressed.
I have the impression that a) the majority of systemd projects are broken by design and b) this is exactly what the LP people wanted.