Hacker News Reader

Comment by dgan | Hacker News Reader

what makes me jealous,is not the double performance ,but the fact that he can handle 13000 files in 2 seconds. i am at my ${corporate} job now, and have about 40'000 json files stored locally on the windows laptop. when I need to load a them, it takes minutes. i am still not sure if i read files wrong or NTFS is a pile of trash when comes to reading many smallish files

View on HN

NTFS is slow, especially when you operate on a lot of tiny files (nobody in the Windows world would do that, you'd always put your tiny data blobs into a bigger container file, e.g. asset files in games), but from my corporate experience, it's mostly the _multiple_ "endpoint security" solutions that bog file system performance down.

It's the reason I so far use a Mac at work, which has its own issues, and a lot of them.

Could be perhaps worth the trouble to try dev-drive/ReFS? [0]

[0] https://learn.microsoft.com/en-us/windows/dev-drive/

NTFS is definitely very slow when it comes to reading many small files. Windows Defender or whatever antivirus you might be using can also further slow this down.

There might me some kind of malware scanner at play.

Maybe it's time to rebuild those with mimalloc, too.

Try excluding that directory from the virus scanner. Also a dev drive might come in handy for this workload

you were right. it didn't occur to me that antivirus would scan everything I open. Moved all files into an excluded directory, reading files is now almost 10 times faster. Thanks kind stranger !

Does ${corporate} prevent you from merging your jsons to a single file with a decent format like parquet?

they don't but that defeats the purpose. i want to be able to inspect the insides, so binary formats are out, i have to easily open & read the content

I don't get how you can't put the content within the merged file and query it but good luck.

because that d be a several hundreds megabytes json :) Anyway with suggestion in another comment, i moved the root directory into an antivirus-excluded location, which gave me almost a 10 times performance win

Reply on news.ycombinator.com