If you use GNU grep on text files, use the -a (--text) option

Actually I recently found that coreutils and ls behave fairly well with funny filenames:

Here is an invalid utf-8 byte and then a valid utf-8 sequence

    $ x=$'\xce\xce\xbc'
    $ touch "$x"

    $ ls
    ?μ

    $ ls --escape
    \316μ

    $ stat *
    File: ''$'\316''μ'

-----

Most command line tools are not aware of stuff like this. For example you can touch "x$ANSI_TERMINAL_CODES" and if you do "bash x??" or "python x??", then your terminal will change color because of the escape codes printed back to the terminal.

I just changed Oil to use a well-defined format I called QSN (quoted string notation):

http://www.oilshell.org/blog/2020/04/release-0.8.pre4.html#t...

It adapts Rust's string literal syntax to express arbitrary byte strings precisely and losslessly. (JSON can't express arbitrary byte strings.)

The QSN encoder does UTF-8 decoding with a specific error recovery mechanism. So it's basically like what ls and stat do, but it's more precise.

(If anyone is interested in QSN, please contact me. I think it's more generally useful in a lot of places. It's something we already do but it's precise like JSON.)

This was changed about five years ago to just keep looking. Which makes things a bit unpredictable in a different way.

Interesting, especially the part about the LINE_MAX. Even though it kinda makes sense, I would never have thought that having a very long line makes a file a non-text file when all characters are 'normal' characters.

And I also think that this would not be a discussion if it was just about NUL. There is something else that grep does not like (escape characters? Invalid utf8? Wrong codepage? I don't know), but this makes the binary detection annoying.

'CWE-117: Improper Output Neutralization for Logs'

That is something probably often forgotten when simply dumping some requests into a log, but at least it should be obvious that the source of the content is untrusted. On the other hand, a log file is a file on your server, so you would probably think of it as nothing dangerous, as everybody has cared about CWE-117, right? ;-)

The speed benefit isn't really a huge factor; working how I'd expect, omitting ignored files, being able to specify file extensions to search, plus simple editor integration. Amazing.

    alias rg="rg --no-heading"

You can disable all smart filtering (gitignore, hidden, binary) with `rg -uuu foo`. That will search the same stuff that `grep -r foo ./` will.

GNU grep doesn't have look-arounds either. It does have back-references, but that doesn't impact searches that don't use back-references.

I don't think there is really a concise way to describe why ripgrep is faster when comparing apples-to-apples. It depends on the queries and the corpus. The primary reasons are that it makes more efficient use of the hardware with algorithms that utilize SIMD.

If you do an apples-to-oranges comparison (i.e., "why is `rg foo ./` so much faster than `grep -r foo ./`), then the answer is pretty easily "because ripgrep uses parallelism and employs smart filtering by default."