Hacker News Reader

Comment by rshm | Hacker News Reader

No date on the post. Probably old and fixed now. Just checked with a random folder, it happily supported 782292 files.

There is mention of 2016.

"The lesson here is that even in 2016, filesystems still are finicky with large numbers of files in a single directory."

Since the root cause is a hash collision in the dir_index code you might just have been lucky and the author of the article unlucky.

They would have had to have been exceptionally lucky - the chance of getting no collisions in a 32 bit hash among 782292 inputs is about 10^-62.

Probably lucky. File names are md5 of the content, i am not sure if it helped on collision on 32 bit md4.

The linked post mentions dir_index default and tune2fs -l lists the dir_index under Filesystem features only.

Not knowing this default behavior, i was considering enabling dir_index explicitly to aid chocking rsync.

EDIT: I ran the reproduction code with 200K instead of 100K just to be sure. There was no problem creating file as well as copying all the files to a different folder. The disk was formatted about 3 months ago and mounted with default options "ext4 noatime,barrier=0 0 0" . On original as well as copied folder `ls | wc -l` still gives 200000.

I think his point still stands. Basically, if you do something far enough outside the norm, you may hit really time consuming bugs.

Although the exception would probably be rare if it used a large enough number, do you think the "copying utility" (whatever it may be) should handle those extreme edge cases? Or is it just "disk full" that's used as a fall-back?

Reply on news.ycombinator.com