Hacker News Reader

Comment by chx | Hacker News Reader

Having a billion files in a single directory versus a more typical tree has its own challenges.

http://be-n.com/spw/you-can-list-a-million-files-in-a-direct...

Possibly, but at the scale discussed in the article, not so much.

  % time ls | wc
  10000000 10000000 78888897
  ls  1.90s user 0.73s system 99% cpu 2.637 total
  wc  0.28s user 0.02s system 11% cpu 2.597 total

ETA: and most of that time is spent sorting the output.

  % time ls -U | wc
  10000000 10000000 78888897
  ls -U  0.53s user 0.29s system 99% cpu 0.823 total
  wc  0.26s user 0.02s system 33% cpu 0.822 total

To my experience (and if only the filename is needed) the "-1" option helps saving some time too. Be aware that output from ls without options differ, if redirected.

time (real user sys) in seconds with 1e6 files in a xfs filesystem. Directory used 27 MB space.

  ls -l > /dev/null     : 49.12 23.90 13.47
  ls -lU > /dev/null    : 46.62 33.11  8.01
  ls > /dev/null        :  6.36  1.97  3.63
  ls -1 > /dev/null     :  2.49  1.83  0.36
  ls -1U > /dev/null    :  0.31  0.18  0.08
  (find)                :  0.77  0.49  0.19

and for 1e6 files in an ext4 filesystem. Directory used 23 MB space. (but as i had no ext4 fs with one million free inodes on a loop device)

  ls -l > /dev/null     : 51.39 34.02 11.39
  ls -lU > /dev/null    : 47.39 30.72 11.09
  ls > /dev/null        :  4.83  3.56  0.65
  ls -1 > /dev/null     :  4.46  3.45  0.50
  ls -1U > /dev/null    :  0.51  0.18  0.26
  (find)                :  1.29  0.76  0.36

Good info. But isn't `-1` implicit when the output is a pipe?

Good reply, you are right. I have even written ("Be aware that output from ls without options differ, if redirected") that it is so but was somewhat fooled by my experiment and the different timings for "ls" and "ls -1" when using XFS.

Of course I was not able to reproduce this difference and assume that I did work on the system and it was not idle.

I have repeated the the tests, on another mostly idle system. (Therefor the timing is not comparable to the first test). This time both filesystems live in a loop device and the benchmarks were run with hyperfine (3 warmup loops, 3 measurements each).

  XFS
  ls -l >/dev/null   Time (mean ± σ):   5.890 s ±  0.394 s  [User: 1.096 s,  System: 3.491 s]
  ls -lU >/dev/null  Time (mean ± σ):   2.254 s ±  0.386 s  [User: 0.648 s,  System: 1.602 s]
  ls >/dev/null      Time (mean ± σ):  542.7 ms ±   4.4 ms  [User: 443.8 ms, System: 98.7 ms]
  ls -1 >/dev/null   Time (mean ± σ):  534.4 ms ±   2.3 ms  [User: 450.1 ms, System: 84.1 ms]
  ls -1U >/dev/null  Time (mean ± σ):  115.8 ms ±   1.8 ms  [User: 80.9 ms,  System: 34.6 ms]
  ls -U >/dev/null   Time (mean ± σ):  121.0 ms ±  14.6 ms  [User: 80.9 ms,  System: 40.0 ms]
  find >/dev/null    Time (mean ± σ):  225.0 ms ±  16.2 ms  [User: 150.1 ms, System: 74.7 ms]

  EXT4 
  ls -l >/dev/null   Time (mean ± σ):  10.493 s ±  0.392 s  [User: 2.495 s,  System: 5.032 s]
  ls -lU >/dev/null  Time (mean ± σ):   8.395 s ±  0.157 s  [User: 0.735 s,  System: 5.049 s]
  ls >/dev/null      Time (mean ± σ):   1.966 s ±  0.010 s  [User: 1.760 s,  System: 0.202 s]
  ls -1 >/dev/null   Time (mean ± σ):   2.014 s ±  0.032 s  [User: 1.798 s,  System: 0.216 s]
  ls -1U >/dev/null  Time (mean ± σ):  221.4 ms ±   1.3 ms  [User: 83.7 ms,  System: 137.6 ms]
  ls -U >/dev/null   Time (mean ± σ):  222.5 ms ±   2.0 ms  [User: 78.9 ms,  System: 143.6 ms]
  find >/dev/null    Time (mean ± σ):  531.0 ms ±   4.0 ms  [User: 377.3 ms, System: 153.7 ms]

That is easily implemented with a double loop, where the first one is a directory (integer hexed with printf '%x') and the second is the files in it

However, that was not the point of my reply

Reply on news.ycombinator.com