Hacker News Reader

Comment by PaulHoule | Hacker News Reader

Depends how you use it.

If you are handling a stupendous number of small files (say doing an npm build) then metadata operations are terribly expensive on Windows because it is expensive to look up security credentials.

Maven is not too different from npm in how it works except instead of installing 70,000 small files it installs 70 moderate sized JAR files that are really ZIP files that encase the little files. It works better in Windows than npm. Npm got popular and they had to face down the problem that people would try building the Linux kernel under WSL and get awful times.

Microsoft knows it has to win the hearts and minds of developers and they believe in JS, TS and Visual Studio code so they’ve tried all sorts of half-baked things to speed up file I/O for developers.

View on HN

No filesystem is great in the lots-of-small-files case, partly simply due to syscall overhead.

There's a reason https://www.sqlite.org/fasterthanfs.html , SquashFS, etc. are a thing, or why even Europe's fastestest supercomputer's admins admonish against lots of small files. https://docs.lumi-supercomputer.eu/storage/#about-the-number...

I guess that's where the "sqlite competes with fopen" part might help.

The reason LUMI is advising against many files is that it uses the Lustre parallel filesystem, which is notoriously bad with small files. See here: https://www.lanl.gov/projects/national-security-education-ce... .

That used to be ReiserFS's claim to fame - tail packing small files. Doesn't seem like any FS has really optimized small file handling since then.

It is not so much the wasted space that bugs me as all the metadata and processing of that metadata.

For instance, the 70,000 files in your node_modules do not need separate owner and group fields (they all belong to the same user under normal conditions) and are all +rw for the files and +rwx for the directories. If you have access to one you have access to all and if your OS is access checking each file you are paying in terms of battery life, time, etc.

On Windows it is the same story except the access control system is much more complex and is slower. (I hope that Linux is not wasting a lot of time processing the POSIX ACLs you aren’t using!)

SquashFS is a (read only) filesystem designed for small files.

Which shows that even if you don't want to call any filesystem great here, they differ vastly in just how bad they handle small files. Windows' filesystems (and more importantly it's virtual filesystem layer including filters) are on the extremely slow end of this spectrum.

Reply on news.ycombinator.com