Show HN: A simple, fast and user-friendly alternative to find, written in Rust

The fish shell is the only one that counts. Fish is life.

> When I'm sshing into a box to figure out whats going on, my toolset is mostly limited to top, find, xargs, awk, grep, df, du etc.

I wrote some of the tools on coldtea's list, and I'm pretty much the same way in that I stick to distro tooling in vanilla setups. But I spend enough time on my local workstation that having "better" (read: creature comforts) is worth it there! But yeah, if you spend most of your time in vanilla setups, then tools not in the standard distro repos are a hard sell.

>I appreciate all these new incarnations of old school tools but end up sticking with the basics regardless. When I'm sshing into a box to figure out whats going on, my toolset is mostly limited to top, find, xargs, awk, grep, df, du etc.

That's why I wrote about it being nice to have them all in distros -- perhaps in a single package like coreutils.

Then wherever you are, they'd be just a "package-manager install rustutils" away.

Generally, if one is not an admin of random hodgepodge of systems (e.g. in big enterprises), then you are:

(1) doing work on your own workstation

(2) administering machines you control the provision (e.g. a person administering a startup's Cloud servers)

(3) doing development on some kind of vm

In those cases you can quite easily install a bundle package and make sure those tools are there.

I, for one, don't go out in unknown systems day after day -- even if we have 100s of servers we manage, we DO manage them.

I love exa and have just mapped it to ls (and ll for exa -l ) .

That way I don't even have to learn the new command (or risk forgetting about ls / be frustrated when I can't use it).

What surprises me is that even after 25 years of the Internet, we cannot bring in new tools into our environment (be it a remote box I ssh into or a trimmed down docker image) on the fly.

The popular package management tools rely heavily on FHS and like to install stuff into directories that require root permission.

Imagine if there was a tool that could download binaries of modern tools from a certain repo and install it to our ~/bin. Imagine if we could use new command line tools as easily as we can download a fully functional complex applications securely into a browser tab just by typing its URL!

A potential solution that would allow you to have your cake (SSH into wherever) and eat it too (have your favorite tools on hand) would be: use a package manager on your hosts that allows for safe installation of per-user packages (that is, such package installation should have absolutely no effect on the system outside of the user’s environment. A logical implication here is that it should be possible to use multiple versions of each package, both inter- and intra-user) and per-user package installation should not require special privileges (i.e. you shouldn’t need to be root if, as per the previous requirement re: safety, per-user package installation has no effect on the rest of the system). Then you could log in and do your work with the best tools you know, rather than being constrained to what your distro (or your servers’ admins) deems useful enough to be installed by default.

It strikes me as an odd choice to accept the least common denominator of tools. As a technologist, that sounds frustrating. And as someone in leadership, I can’t imagine hamstringing my team by making them work within an environment that won’t let them use the best tools for the task at hand.

That's the kind of idea that lead to Joey Hess's "moreutils".

https://joeyh.name/code/moreutils/

grep came, then ack-grep, then silver-searcher and a few others, finally ripgrep. For most purposes, ripgrep (the one written in Rust) is the best of all of these tools, typically being the fastest and also the most Unicode-compliant, with the main downside being its non-deterministic ordering of output (because it searches files in parallel).

The only reason for using ack these days is if you’re already invested in a Perl workflow and can use some of the Perl magic powers, and don’t care about comparatively poor performance. That’s not many people.

Pup deserves a mention too.

https://github.com/ericchiang/pup

I've tried ag and ripgrep. ripgrep is better.

That's basically what's happening. AFAIK, ack kinda started it all by observing, "hey, we frequently have very large directories/files that we don't actually want to search, so let's not by default." Depending on what you're searching, that can be a huge optimization! Tools like `git grep` and the silver searcher took it a step further and actually used your specific configuration for which files were relevant or not. (Remember, we're in best guess territory. Just because something is in your .gitignore doesn't mean you never want to search it. But it's a fine approximation.)

Was this a thing back when the BSD and GNU greps were being developed? Was it common for people to have huge directory trees (e.g., `node_modules` or `.git`) lying around that were causing significant search slow downs? Not sure, but it seems to have taken a while for it to become a popular default!

There are of course other tricks, and most of those are inspired by changes in hardware or instruction sets. For example, back in the day, a correctly implemented Boyer Moore was critical to avoid quadratic behavior by skipping over parts of the input. But that's lessish important today because SIMD routines are so fast that it makes sense to optimize your search loop to spend as much time in SIMD routines as possible. This might change how you approach your substring search algorithm!

... so what's my point? My point is that while sometimes optimizations are classic in the sense that you just need to change your heuristics as hardware updates, other times the optimization is in the way the tool is used. Maybe you can squeeze the former into existing tools, but you're probably not going to make much progress with the latter.

I remember when I first put out ripgrep. Someone asked me why I didn't "just" contribute the changes back into GNU grep. There are a lot of reasons why I didn't, but numero uno is that the question itself is a complete non-starter because it would imply breaking the way grep works.

Really cool, I've needed this quite a number of times. Thanks for the tip!

> Pretty much PostMan (that a lot of ppl use for no reason)

I can't see anywhere in the httpie docs about capturing variables from and/or running tests on the HTTP responses - it's an extremely powerful feature of Postman when you're debugging non-trivial HTTP flows.

If you're grepping binary files, then even with grep, you need to pass the `-a/--text` flag. ripgrep has the exact same flag.

Protips: `rg -u` stop looking at `.gitignore` files. `rg -uu` searches hidden files/dirs. `rg -uuu` searches binary files.

> Another elitist aside: I can usually tell that I won't want to use a tool when it has a mac os install option first under the downloads section on its page.

This is only because it's so much harder to support every possible Linux distro. I actually don't use macOS on my own, but 'Homebrew' seems like a really nice package manager. (Also, the first install option is via "cargo" which is platform independent.)

But in fact it does not seek to mirror _any_ of find's powerful functionality.

Maybe advertize it as "a better ls -R | grep -i"?

It took me a while to figure out `-o`. You need to use `\(` `\)`

    find . -type f \( -name \*.c -o -name \*.cc -o -name \*.cpp \)

Quick FYI: xargs is not the same as execdir; execdir executes the command in the same directory as the file. And to account for spaces in filenames (yay, OSX), your command starts getting a bit more complicated: fd -0 .py | xargs -0 stat

Or parallel, which is slightly trickier to implement than xargs/-exec.

xargs does have limits on input length. On some platforms they aren't terribly high. Sound like -exec is potentially coming though.

Edit: Swore I had gotten ARG_MAX complaints out of xargs in the past. Replies are correct though. I'm wrong here...xargs adjusts on the fly. Perhaps long ago, or some obscure use of xargs?

find ... -ok ... {} \; is easier to type than find ... -print0 | xargs -0 -p -I{} ... {}

It's not even slower (and it's safe beyond the argv limit) if you use "+" instead of the semicolon.

  find -type f -exec sed -i s/foo/bar/ {} +

This only invokes the command twice for 100k files (assuming we can pass in 64k arguments to sed)

You need it if you want to modify the directory structure that find is traversing, so as to affect subsequent traversal, eg http://www.chiark.greenend.org.uk/~pmaydell/find/primes (cheatsheet here: http://www.chiark.greenend.org.uk/~pmaydell/find/primes-comm...)

Admittedly I wrote that nearly 20 years ago so I guess it doesn't invalidate your point :-)

What about commands which only take a single file as input? Is xargs -n1 noticeably faster than -exec? (And if it is, is that actually worth it for clarity's sake?)

see a detailed discussion here: https://unix.stackexchange.com/questions/321697/why-is-loopi...

especially this quote:

> find's business is evaluating expressions -- not locating files. Yes, find certainly locates files; but that's really just a side effect.

> I'm curious why you (or anyone) uses -exec?

I already know how to use it. It works.

> It's often painfully slower than piping through xargs

I've tried it. It has different semantics. First attempt always leads to errors and doing the wrong things. I always have to come up with some workaround to make things work as I expect they would (i.e. how -exec does things by default).

This lack of confidence also means I would never dare using xargs for anything remotely destructive (like rm).

What reason do I have not to use -exec? Why swap a perfectly perfect hammer for a broken screwdriver?

Besides: IMO the timesaving aspects of find is not strictly tied to how fast find runs. It's tied to what I can automate.

In my opinion, xargs has really niche use cases, and -exec is way more fit for general use.

- There's the issue of command line length, pipe too many files and xargs will just fail.

- There's predictibility. With -exec you get to see the entire command line structure, what is escaped, what is not, what names go where. With xargs, a couple of badly named files and you are out of luck.

- Find's syntax is just more general. Xarg is limited to pushing parameters into a script, while -exec can run whatever scrit you want, with single or multiple parameters, or flags, or whatever.

Performance is almost never an issue.

YMMV? Having used find -exec for 20+ years, I find it much more difficult to convince myself to switch to tools like this, and just simpler to use -exec. That, and the extra spawn - and potential for disaster - that putting a | into the mix can incur ..

I couldn’t use neither thing without man page for the first time. And remembering -name -iname -type* and -I -s comes only with practice.

* does fd find files by type, mtime, depth, logical operators? I missed that in readme. If not, maybe “just pattern” argument is not too fair at all.

I agree, but replacing find would be very hard, not only does people except it to work in a certain way, but tons of shell scripts also.

any plans on implementing the exec command from find in fd?

You could always add a --back(wards compatible) parameter that caused it to parse every other parameter like a `find` :)

The most annoying part of find, for me, is stuff like exclusions, with `-prune -o ... -print`. Something like rsync's `--exclude` would be great.

It is interesting if

  alias fd=“function _fd(){find . -iregex ‘.*’$1’.*’};_fd”

would suffice for these 99? (not tested, but hope you got the idea)

Then you have a different usage pattern than I do. For simple name globbing, I usually just use shell globs, e.g.

  ls src/**/*_test.go

It might be marginally slower than find or FD, but it usually doesn't matter because I usually deal with a small number of files.

Did you do a benchmark with `--threads 1`? I suspect most of the speedup is coming from that as the the listing of the files is likely to be IO bound (or at least bound by calling into the kernel to read the directory listings).

That has certainly been my experience in the past when experimenting with this sort of thing, that more threads makes a lot of difference.

You're right that fzf doesn't replace find. Fzf is just an interactive, filtering tool. You can pipe anything into it. find is a common way of populating it with data.

I suspect the GP meant that fzf replaced the pattern 'find |xargs $binary' for small interactive use cases. It's much nicer to do '$binary <invoke fzf>'. I use fzf the same way with key bindings to select directories and files powered by find.

It's not a find replacement at all, even!

fzf just actually calls find by default, so a naked call to fzf will actually give you the same results as find | fzf.

Search specifically for directories, that's find -type d | fzf.

Yes. For simple searches, the main reason is that 'fd' walks the directory tree in a multithreaded fashion (thanks to the 'ignore' crate).

Should have been more clear heh. There was a benchmark that didn't use regex and seemed to turn off the ignore files feature. Im not familiar with rust or those crates Im assuming that's what those are there for?

The docs don't say: is this meant to replace Busybox or the actual GNU coreutils? As in, all GNU features and flags (which are generally much more comprehensive and IMO nicer than plain old POSIX features).

You can't install this with cargo. Una is a Haskell program; you can get it with `stack install una`.

You could just use bsdtar from libarchive, which handles all the compression and archive formats automatically without additional tools.

https://github.com/libarchive/libarchive/wiki/LibarchiveForm...

GNU tar is so limited, but unfortunately still the default on most systems.

Awesome. Sorry I should’ve dug a little deeper before commenting. Nice job on the multithreading also!

> using bench for some nice statistics

'bench' turns out to be this tool: https://github.com/Gabriel439/bench

It seems to be quite useful, and I was not aware of it, thanks! Would probably be nice to have it packaged in distributions...

I am not sure if I find that a smart default as I frequently exclude generated files from git repositories and when want to find something I do not know why I would not like to find such a file if my pattern matches it.

I would prefer to let the switch enable the .gitignore logic, but as I don't know the authors use-case, theirs might be valid too.

Aye, that's because being written in Python, C or Go is - in my eyes - a strike against the project in an absolute sense, because it will be more buggy and difficult to maintain due to lacking the functional-programming abstractions and powerful static-type system of a good ML descendant such as Rust or OCaml. (Or of Haskell, but it tends to lack the practicality of the ML family.)

So of course one does not trumpet that one's project is written in Python, C or Go.

n.b. I said "absolute sense" because all of this this is, of course, inapplicable when searching for libraries specifically for a language such as Python, C, or Go.