Hacker News Reader

Comment by ck2 | Hacker News Reader

This is a nasty hack where it is counting the output from find.

What do you do when find cannot handle the sheer number of files, argument too long from the linux hardcoded MAX_ARG_PAGES

We need a dedicated utility that counts inodes.

View on HN

That is not at all correct.

The output from find is a list of filenames separated by newlines. The wc command is reading from stdin and counting those newlines[1] -- wc itself has just two arguments in its argv array here ("wc", "-l"). You will not exceed its argument list. Further, it is not at all a hack to count the number of files this way; rather using pipes to compose functionality is the Unix way[2].

But fwiw, if you want a count of the inodes in use in the entire filesystem, you can get that directly from "df -i".

In any case, as an exercise to the reader, I encourage you to grab the source to find and add the -count output option you desire.

[1] Technically a filename can contain newlines, so this would throw off the count. The fix for that would be to use find's -print0 output option and then pipe to something which can count on the nulls. In practice, you're unlikely to have such filenames.

[2] cf. http://harmful.cat-v.org/cat-v/unix_prog_design.pdf

I think you are confused — at no time are any files passed as arguments in the command "find . | wc -l".

Also, to count the number of files in a directory, you need to do readdir(), which will get you the names. Then, to see if any of them are directories to be recursed into you need to stat() the names individually. This is all "find" does. How, I'd very much like to know, is this "a nasty hack"?

I am not saying find is outputting files, I am saying it is decoding the full table entry for each file and formatting it, then passing that listing as an entry to wc. Which has to be a bunch of overhead that adds up when you have hundreds of thousands of files to count.

All we want from find is "count the number of inode entries in this tree branch that are type f". No formatting, no output, just count. That has to be much more efficient.

You are going to need to rely on the filesystem authors to write a tool to traverse their inode trees directly, or probably write something like that yourself.

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux....

OMM, looking at 700k files on a spinning disk, I'm getting an uncached run at 1m13s, and a cached run at 1.2 seconds.

That's "time find . | wc -l"

It's going to be disk limited unless you're in cache.

Switching to "time find . -printf '1' | wc -c" lowers overhead to .7 seconds, so roughly 1us/file.

There is a more important benefit to using the -prinf '1' approach - which is that it won't miscount the result if any of the filenames contain newlines.

In this invocation find and wc both have two fixed arguments, you're never going to hit MAX_ARG_PAGES. The output from find is passed to wc over a pipe, and to my knowledge there's no upper limit on how much data you can transfer over a pipe. I've piped huge disk images.

The only argument to his find command is '.', so where is the concern about argument list?

Reply on news.ycombinator.com