Hacker News Reader

Comment by wayneftw | Hacker News Reader

Wow, so maybe instead of a node_modules folder, npm should use a node_modules.db?

This sounds like a fantastic idea. You would go from 4000 directories and 50000 files to a single file with b-tree indices, hashes, etc. You could probably get a 100x speedup over the current approach.

Even better would be not to use it entirely - it'd solve far bigger problems with node.

Here you go: https://github.com/guardianproject/libsqlfs

Thank you, I was reading the comments exactly to figure out if there was a FUSE sqlfs module. TL;DR this URL describes that, yes, there is. Why is this relevant? Because this way, you keep the flexibilities of a filesystem and, as a consequence of it being able to function like a filesystem, you keep the "power of Unix/Linux" ie. all your <asterisk>nix [1] tools keep working.

[1] Calling it Nix confuses with NixOS...

So call it *nix? It takes two asterisks two turn on italics between them.

Cheers. Yeah, I guess it triggered cause I used two asterisks in my text since originally I used the term twice.

well, it was *nix before nix (the package manager) existed. So I think Nix, as good as it is, stuffed up with that name.

There's already support for accessing zip and tar archives as filesystems (also see ASAR), which doesn't incur the overhead of SQL evaluation. At least on Windows, this would result in a huge speedup (just because you're avoiding antivirus latency). I don't expect it to help Mac or Linux that much (as their filesystems are comparatively much more efficient with larger numbers of small files).

I had a problem with zips, they can be only 232 bytes = 4.3 GB. The Zip64 format fixes this, however few libraries support it.

That sounds like an upside. Force people to keep their hypothetical node_modules.db "lean and mean"!

That would be amazing!

To work though, you'd need some kind of interface to actually access the contents. I guess that's doable, but not ideal.

Someone else posted libsqlfs, which is FUSE Sqlite file system, and should do the job...not exactly plug-and-play though, and presumably wouldn't work on Windows (maybe on WSL tho?)

WSL1 is terrible for I/O (which I suppose this is). WSL2 should be adequate.

Perhaps Dokan? Dokan has a FUSE wrapper [1]

[1] https://dokan-dev.github.io

Not sure if /s :)

With a FUSE filesystem, you're still incurring the syscall overhead that's causing this performance disparity in the first case, plus now the kernel has to context-switch into the FUSEing process. So what would be 2 context switches for each syscall (program -> kernel -> program) now is 3 or 4 context switches (program -> kernel -> FUSE -> (kernel ->) program).

FUSE has some amazing usecases, but performance is none of them.

Erm, no, no sarcasm, but in truth I know very little about FUSE, beyond it being an API for userland file systems.

I'm also approaching this as primarily a Windows user - a lot of the overhead on Windows is access checks, so my thinking is these would be implemented as noops.

It’s kind of the idea behind Yarn’s Plug’n’Play: https://yarnpkg.com/en/docs/pnp

Reply on news.ycombinator.com