Hacker News Reader

Comment by vetinari | Hacker News Reader

It is not just fat32 and overhead up to cluster size; once I had 800k tiles in a single directory on ntfs.

It was unusable. The only thing that I was able to do is to tar it up and move to a machine with xfs, where I was able to sort it up into more balanced subdirs and then move it back (for processing using windows-only tool). Just tarring that single directory up took several days.

View on HN

It's not just ntfs. I tested this a few months ago in a pretty unscientific manner using ~50 million files in one directory.

btrfs was unusable (not only that particular directory, but the whole filesystem became noticeably slower).

ext4 was ok. xfs didn't break a sweat. I don't recall any practical difference when compared against a nested tree like

  ├── aa
  │   ├── aa
  │   │   └── aaaaf3ee5e6b4b0d3255bfef95601890afd80709
  │   ├── ab
  │   └── ac
  └── ab
      ├── aa
      ├── ab
      └── ac

Reply on news.ycombinator.com