Hacker News Reader

Comment by CaliforniaKarl | Hacker News Reader

This was written by my coworker, Kilian Cavalotti. We have two main file stores: $SCRATCH is a Lustre cluster, which is extremely performant but doesn't do well with lots of inodes. And $HOME is a multi-node Isilon, which can handle all the inodes, but is not as performant. And we have users who sometimes like to put many files in single directories.

You can find more information about us at https://srcc.stanford.edu

And you can find more information about Sherlock at https://www.sherlock.stanford.edu

View on HN

Thanks for the additional information but sometimes I wonder that 15k+ files count as 'many' nowadays. I mean, an off-the-shelf Canon DSLR produces 20k files per directory (10k JPG + 10k raw). Nevertheless, I did some test your approach on my PC:

On a NFS mounted on my PC your LS_COLORS tweak actually degrades the performance. Without modification, it takes 0.5 seconds to list a directory (on a slow consumer grade HDD) with 14448 files and after setting

  export LS_COLORS='ex=00:su=00:sg=00:ca=00:'

it takes 4.4 seconds.

However, listing a local (SSD) directory with 15k files takes just about 0.12 seconds and gets faster after settings the variable to 0.06 seconds.

For all tests I drop the caches before testing, e.g.:

  echo 3 > /proc/sys/vm/drop_caches; time ls --color=always /mnt/nfs/many | wc -l

"On a NFS mounted on my PC your LS_COLORS tweak actually degrades the performance"

Run "strace -o logfile ls --color=always" and diff the two logfiles.

Well, this looks rather strange. As expected, settings the variable causes large changes as it removes a lot of lstat calls and also removes the color codes from the writes. So there seems to be no obvious explanation, why it becomes slower when doing less...

  # du -sh logfile_nfs*
  1.7M    logfile_nfs
  668K    logfile_nfs_exported

However, the strace -c output looks kinda different:

Without modifying the variable: https://bin.disroot.org/?149fa91c08b27312#0w9O6BAWNEEC4SUXEb...

With setting the variable: https://bin.disroot.org/?149fa91c08b27312#0w9O6BAWNEEC4SUXEb...

You pasted the same link for both cases. Can't tell what's going on.

Oh sorry, my fault. The second link should be

https://bin.disroot.org/?56f3dcb618240df0#x3StRGZZgSzM0UJ0BK...

Plus/minus what? Note that caching on the server can be relevant, and you can't control that from the client. That's one thing that requires care for performance measurements on networked filesystems.

Sounds as if you don't trust my statement. Sure there is some variance, but when one is measured in seconds and the other in the tenth of a second it should be clear that it is not a coincidence...

But for completeness, I wrote a few loops which should answer your question:

=NFS=

Normal: https://bin.disroot.org/?8967a0f34b26e512#aTYmbyESfeuqAXS802...

With exported Variable: https://bin.disroot.org/?c405d7aa74b50ee0#KverJYJEzNhct7CmdW...

-----

=Local=

Normal: https://bin.disroot.org/?b94e1d1f58e3edb7#tKszD/tjvwBwepJLun...

With exported Variable: https://bin.disroot.org/?d3ac83f1dff9e767#WVODjGnvL1QbOJdWPp...

I also tried dropping the cache on the NFS server but that didn't seem to have a major effect on the performance (probably because reading a 15k file index from a local disk doesn't take that long after all).

I don't understand the "lots of inodes" comment, but any filesystem limit on inodes isn't relevant to this, which is just due to RPCs to the metadata server(s). (ls -l is actually worse than coloured plain ls because the size data are on the OSS, not the MDS.) The canonical advice for general performance is to keep directories reasonably small on Lustre due to possible lock contention, but I don't know the circumstances for which that's actually relevant. [Metadata operations, such as large builds and tars, are typically slower on our Isilon than on the Lustre filesystem, which has no serious tuning as far as I know.]

> The canonical advice for general performance is to keep directories reasonably small on Lustre due to possible lock contention, but I don't know the circumstances for which that's actually relevant.

I helped a researched debug a Lustre performance issue a while ago. Each job was nothing special, read a few files (maybe a few GB total), do some (serial, no MPI or such) calculations taking maybe 10 min or so, produce output files, again a few GB. No problem, except when the person ran a several hundred of them in parallel as an array job the throughput per job dropped to a small fraction of normal. Turned out that all the jobs were using the same working directory. Slightly tweaking the workflow to have per-job directories fixed it.

what is an Isilon? thanks

https://en.wikipedia.org/wiki/Dell_EMC_Isilon

And more on the product in particular: https://en.wikipedia.org/wiki/OneFS_distributed_file_system

Disclaimer: I work for Dell.

Hyperscale Network Attached Storage!

Disclaimer: I work on Isilon

Why was iSCSI support deprecated? I recently developed an interest an iSCSI, which makes me curious; not an Isilon user, though.

My guess would be that Isilon/OneFS is designed for file storage, instead of block storage, so having a block storage layer on top (like iSCSI) seems a little ungainly. Instead I expect Dell/EMC would prefer you investigate the VNX platform!

Edit: By "file storage", I'm talking about storage mounted using protocols like SMB and NFS.

That's actually a disclosure, not a disclaimer.

Clustered storage.

Tell your colleague I think emojis make the article look like it was written by a 12 year old.

IMO. I guess I'm just an old fart.

I think they add a bit of emotion to the text written.

It's not because it has colours that it's not a serious article.

It is how most millenials write nowadays

It’s how people trying to write like millennials write

Postfix emoji, eurgh

Even if it were, maybe you should have more respect for 12 year olds.

IMO. I guess I'm not an ageist.

Reply on news.ycombinator.com