Hacker News Reader

Comment by _urga | Hacker News Reader

Removing `node_modules` would be fantastic, but not at the expense of native modules:

From the PDF:

"On the long term we believe that post-install scripts are a problem by themselves (they still need an install step, and can lead to security issues), and we would suggest relying less on this feature in the future, possibly even deprecating it in the long term. While native modules have their usefulness, WebAssembly is becoming a more and more serious candidate for a portable bytecode as the months pass."

WebAssembly is not "more and more a serious candidate" to replace native modules.

The issue with post-install scripts needs a better long-term solution, but simply deprecating native modules is not it.

View on HN

I mentioned it in another answer, but we'll eject the packages that require postinstall scripts - so they will work as before, but will take extra time to be installed.

As for wasm, I'm curious to hear what you think isn't good enough. I think the two main issues are garbage collection and dynamic linking, and there's ongoing work on them to fix them.

"I mentioned it in another answer, but we'll eject the packages that require postinstall scripts"

Sure, but you also mentioned in the PDF that the other solution was just to deprecate native modules in the long term, and that's not acceptable.

"As a data point, we encountered no problem at Facebook with adding --ignore-scripts to all of our Yarn invocations. The two main projects we’re aware of that still use postinstall scripts are fsevents (which is optional, and whose absence didn’t cause us any harm), and node-sass (which is currently working on a WebAssembly port)."

I think you're too quick to sacrifice native modules because you're not really using them.

I did work on the asmjs/wasm bindings for Yoga, Text-Buffer, and a few other projects, so I'm biased :)

Anyway, regardless of my own long term feelings, rest assured that postinstall scripts will be supported as well as they are now.

Isn't native interoperability the entire point of Node.js though?

Filesystem access etc. has to, at some point, cross the barrier from VM to native code. Sure Node.js has file-system access (and other stuff) built in, but willingly moving in a direction that's non-extensible seems odd.

I know you've since said you'll just pull native modules out of the cache, and I'm perfectly happy with that. However, please don't overlook the importance of native interoperability because the code you tend to write abstracts this away from you. Native interoperability is what makes Node.js.

Native code has two benefits over JavaScript: it's faster, and it can call the OS. WASM is faster, but it can't call the OS. It cannot replace all uses of native modules.

Wasm on the browser can't call the OS, but I don't assume that to be the case on Node, just like it isn't for JS.

JS (and eventually wasm) calls the OS via requiring native modules, which is why we're concerned about keeping them around.

Calling the OS doesn't require native modules, it should be supported via a native cffi in node, like in Python.

Are you describing something you wish that node had, or something that actually exists? As far as I'm aware, native modules are the FFI for node, and that every package that involves presenting an FFI to the developer is using a native module to do it.

Node doesn't have it natively like Python does, but it exists:

https://github.com/node-ffi/node-ffi

Yeah, that's a native module. If you look at the C source code in "src", the "NAN_METHOD" and stuff are Node native module macros. node-ffi will not work if they remove native modules.

I don't think WAM achieve native performance yet. You need to copy data in and out of it instead of zero-copy. WASM can't link to native system libraries. WASM can't access most hardware.

It's very neat, but I can't see it fully replacing native modules soon.

Wasm is designed to deal with the VM sandbox, which if you're writing a server is fantastic compared to actual native code as there's better security properties.

It's not fantastic if you're working on an Electron app where there might be one specific feature that requires a native module in order to hit an OS API that you can't do from a browser sandbox (and that Electron itself doesn't expose).

Rather than pursue total deprecation/eventual removal of postinstall/native modules, I think a package.json "allowNativeDependencies": "never" | "devOnly" | "always" option that defaults to "devOnly" if not specified is the way to go.

i would say make it a boolean flag, default to false, with a comment that says "this must be enabled for packages which use the local filesystem or OS APIs". if wasm delivers on its potential, OS-level interactions would really be the only valid use case (and legacy code, which should be considered a little bit inherently unsafe anyway).

perhaps yarn could even develop official packages which include native code that wraps all the most relevant OS APIs, and the default setting of the flag would allow only those native packages.

this is all pretty far down the road anyway.

Almost all the native modules I come across are about accessing system calls not available from within the sandbox. Only a few are there for speed, and since native calls add quite some overhead, they are only useful in specific use cases.

I didn't dive deeply enough, but I'd assume they could just fallback to classic node_modules behavior for the native packages, right? You'd not actually delete the node_modules directory then but still get most of the speed benefits.

EDIT: just saw a comment by the author that says roughly this: https://news.ycombinator.com/item?id=17977986

So, native packages will keep on working.

Has WASM become the new bittorrent/cryptocurrency/ML that's going to solve any and all problems?

Reply on news.ycombinator.com