Hacker News Reader

Comment by teddyh | Hacker News Reader

I think you are confused — at no time are any files passed as arguments in the command "find . | wc -l".

Also, to count the number of files in a directory, you need to do readdir(), which will get you the names. Then, to see if any of them are directories to be recursed into you need to stat() the names individually. This is all "find" does. How, I'd very much like to know, is this "a nasty hack"?

View on HN

I am not saying find is outputting files, I am saying it is decoding the full table entry for each file and formatting it, then passing that listing as an entry to wc. Which has to be a bunch of overhead that adds up when you have hundreds of thousands of files to count.

All we want from find is "count the number of inode entries in this tree branch that are type f". No formatting, no output, just count. That has to be much more efficient.

OMM, looking at 700k files on a spinning disk, I'm getting an uncached run at 1m13s, and a cached run at 1.2 seconds.

That's "time find . | wc -l"

It's going to be disk limited unless you're in cache.

Switching to "time find . -printf '1' | wc -c" lowers overhead to .7 seconds, so roughly 1us/file.

There is a more important benefit to using the -prinf '1' approach - which is that it won't miscount the result if any of the filenames contain newlines.

You are going to need to rely on the filesystem authors to write a tool to traverse their inode trees directly, or probably write something like that yourself.

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux....

Reply on news.ycombinator.com