Hacker News Reader

When I looked ag https://github.com/ggreer/the_silver_searcher was more featureful than ripgrep yet it's always ripgrep that is mentioned. :/

weird right? I don't know what ripgrep has, maybe it's just the name? the fact that you call `ag` but it's called `the_silver_searcher`?

From my side, I knew about both `ripgrep` and `the_silver_searcher` but I will openly admit I've lost faith in C and C++'s abilities to protect against memory safety errors.

Thus, Rust tools get a priority, at least for myself. There are probably other memory-safe languages but I haven't had the chance to give them a spin like I did with Rust. If I found out about them then I'll also prefer tools written in them if there's no Rust port and if the alternatives are C/C++ tools.

This is what I suspected: Rust zealotry. We are searching for a string. Language matters to the guy writing it but the user? Nah. There's no security hole here.

Who's the zealot here? The guy who doesn't want to risk and openly states so, or the guy proclaiming there's no risk, even with ample historical evidence for the opposite?

I don't accept your labeling, especially when it's so egregiously misguided.

Also, you're clearly seeing what you want to see.

Ag is shorthand for argentum which in turn means silver.

https://en.wikipedia.org/wiki/Silver

Code search on your own codebase is generally not a place where memory safety shines.

When it's about security, I try not to pick and choose.

Plus a memory error might lead to scary consequences (like a random script getting elevated privileges).

Finally, and last I checked, `rg` is quite featureful and I've never felt constrained by it. So for me it's a win all around.

This is kind of my point: security requires a threat model, and you don't really have one. Rust has a lot going for it, and it does hold promise in improving the security of a lot of critical software. But in this case, it's not really doing that, so it's kind of misleading to say it is meaningfully doing anything for security.

I agree that in the case of grep-ing in the terminal the odds of covering your butt well enough by using a Rust tool are super slim.

That being said, there are powerful adversaries of anonymity and the right to personal data out there -- and security in depth is what works best against them. There's no one UltimateSecuritySolution™; there are many small ones that we layer on top of each other so we don't allow even a smidgen of air to pass between the cracks.

But yeah, I am paranoid. I am gradually preparing myself to move from macOS to Linux and even though I am not a criminal and never will be, I'll still make a heroic effort to make the odds of any foul play against me practically zero. (And that's why I will start using the userland Rust tools alternatives as well.)

I'll concede that in my case the biggest impact would probably come from running Chrome in a jail, and not from using `rg` vs. `ag`. That much is true, yep.

Yeah, it's kind of a shame in this case. There's tons to talk about in the area of where Rust shines here ("makes concurrency easy", "provides easy access to fast algorithm libraries", etc.) but security is just not really one of those points.

I don't disagree. I am just happy to point out that Rust increases security (since most security vulnerabilities I see reported are buffer over/under-flows or other memory safety mishaps). Rust definitely does not solve everything in security. You can still open yourself up for an elementary replay attack if you're not careful -- like I did just a few days ago.

ripgrep is noticeably faster than ag. I'm not sure what features you mention that ripgrep is missing, but I've been plenty happy with it for basic grepping around. I'm sure it's also partly because BurntSushi, the author of ripgrep, is reasonably active here.

Some fun benchmarks, and a fair description of how text search works, can be found in this 2016 post https://blog.burntsushi.net/ripgrep/

> ripgrep is noticeably faster than ag.

Since both are instant on every codebase I care about, not so sure about that. These tools were always chiefly I/O bound , SSDs have torn that down especially now.

But for features: there's no lookaround in the regex.

These tools are I/o bound, but not disk bound. I often perform multiple searches in a row, and all but the first will hit the disk cache.

They are only I/O bound when searching data that isn't in cache. It's often the case that you're searching, for example, a repository of code repeatedly.

> Since both are instant on every codebase I care about, not so sure about that.

It is very possible that there is no performance difference between ripgrep and ag for your use cases, but that does not mean there isn't a performance difference between ripgrep and ag. For example, in my checkout of the chromium repository:

    $ time rg -c Openbox
    testing/xvfb.py:2
    tools/metrics/histograms/enums.xml:1
    ui/base/x/x11_util.cc:1

    real    0.448
    user    2.593
    sys     2.490
    maxmem  77 MB
    faults  0

    $ time ag -c Openbox
    ui/base/x/x11_util.cc:1
    tools/metrics/histograms/enums.xml:1
    testing/xvfb.py:2

    real    2.302
    user    2.996
    sys     10.462
    maxmem  15 MB
    faults  0

> But for features: there's no lookaround in the regex.

That's not true and hasn't been true for a long time. ripgrep supports PCRE2 with the -P/--pcre2 flag. You can even put `--engine auto` in an alias or ripgreprc file and have ripgrep automatically select the regex engine based on whether you're using "fancy" features or not.

In general, I also claim that ripgrep has far fewer bugs than ag. ag doesn't really get gitignore support correct, although if you only have simple gitignores, its support might be good enough.

It's not weird because ripgrep has had more features than ag for a long time. Originally ripgrep didn't have multi-line support or support for fancy regex features like look-around, but it has both of those now. It also has support for automatic UTF-16 transcoding, preprocessors for searching non-text files and overall less buggy support for gitignore. (Look at ag's issue tracker.)

And then there's also the fact that ag isn't that great as a general purpose grep tool. It really falls over in terms of speed when searching, say, bigger files:

    $ time rg 'Sherlock Holmes' OpenSubtitles2018.raw.en | wc -l
    7673

    real    1.475
    user    1.115
    sys     0.356
    maxmem  12511 MB
    faults  0

    $ time ag 'Sherlock Holmes' OpenSubtitles2018.raw.en | wc -l
    7673

    real    20.276
    user    19.850
    sys     0.413
    maxmem  12508 MB
    faults  0

A lot of people like to comment and say, "well ag is fast enough for me." Well, OK, that's fine. But if you're wondering about why other people might mention ripgrep more, well, maybe it isn't just about the way you use the tools. For example, if you only ever search tiny repositories of code, then you aren't going to care that ripgrep is faster than ag. Which is perfectly reasonable, but it should also be reasonable to be aware that others might actually search bigger corpora than you.

ripgrep regexp has lookaround? The linked docs still doesn't say that.

https://docs.rs/regex/1.5.4/regex/#syntax

> This crate provides a library for parsing, compiling, and executing regular expressions. Its syntax is similar to Perl-style regular expressions, but lacks a few features like look around and backreferences.

Edit: I see. It can use pcre2 but it's a build time option which of course Ubuntu has off.

> Edit: I see. It can use pcre2 but it's a build time option which of course Ubuntu has off.

That's unfortunate. Perhaps a bug report to the packagers is in order? Archlinux enabled it: https://github.com/archlinux/svntogit-community/blob/0dc033f...

https://bugs.launchpad.net/ubuntu/+source/rust-ripgrep/+bug/...

I still haven't seen anyone elaborate on what features ag has over ripgrep. Can you explain what you like about ag that ripgrep currently doesn't do?

Which features does it have that ripgrep doesn't?

In an attempt to remove some of my ignorance, I tried this: (MBA, M1)

  time rg Query src > /tmp/junk.rg
  rg Query src > /tmp/junk.rg  0.01s user 0.04s system 156% cpu 0.037 total
  time pt Query src > /tmp/junk.pt
  pt Query src > /tmp/junk.pt  1.44s user 10.80s system 220% cpu 5.545 total
  time ag Query src > /tmp/junk.ag
  ag Query src > /tmp/junk.ag  4.32s user 9.82s system 204% cpu 6.918 total

  ls -l /tmp/junk\*
  -rw-r--r--  1 sramam  wheel  16492080 Nov 10 16:05 /tmp/junk.ag
  -rw-r--r--  1 sramam  wheel  16492080 Nov 10 16:04 /tmp/junk.pt
  -rw-r--r--  1 sramam  wheel     27477 Nov 10 16:04 /tmp/junk.rg

This is a TypeScript module, turns out ripgrep correctly ignores the sourcemaps for the improved performance. I like tools that are smart by default.

(EDIT: formatting)

I think rg isn’t ignoring sourcemaps to be fast. It tries to ignore based on .gitignore and similar files, but maybe the sorcemaps are ignored for being binary.

I thought so too. The documentation for all state they honor .gitignore. In my case, the compiled code is committed.

My query was for a generic term "src" - so that likely trapped the file-path embedded within the long lines of a source-map. Hence the massive file-size difference.

(I did inspect the files to validate)

The main goal of respecting gitignore is generally to decrease noise in search results. But, this does of course also have the effect of improving performance in a lot of cases.

Okay, call me lazy. Not for ripgrep, installed it early from sources. However your fdfind (fd) mention made me curious. Thought I give it an apt install on Ubtunu 20.04 LTS but dead end. So perhaps in debian-stable but not in Ubuntu. Just saying, so I can feel less out of the loop ;)

Upstream says Ubuntu's package is 'fd-find' (and the executable is 'fdfind' -- both renamed from upstream's 'fd' because of a name collision with an unrelated Debian package. If you don't care about that one, you can safely alias fd="fdfind").

https://github.com/sharkdp/fd

(I've edited my first comment in response to this reply: I originally wrote "fdfind". (For a comment about regexp tools, this is a uniquely, hilariously stupid oversight. Sorry!)).

Well, even its late here, I could still have enough creative energy to insert the minus in there ... yeah, thanks for that. So now I feel really behind because sure, it's packaged in Ubuntu 20.04. Thanks again.

There is also ouch, a fast multi-format compressor/decompressor written in Rust

https://github.com/ouch-org/ouch