Node's nested node_modules approach is basically incompatible with Windows | Hacker News Reader

Node's nested node_modules approach is basically incompatible with Windows

github.com

63 pointstsing58 comments

Know the biggest issue I've had with Node (well not me, but my colleagues)? That modules have a habit of using symlinks, which break horridly across Vagrant/VirtualBox shared folders. We ended up fixing it somewhat by using the --no-bin-links option on `npm install` and swapping over to SSHFS, but I really wish module writers would pay more attention to Windows... one of Node's strengths is that it works pretty darned well on Windows (just like how XAMPP made PHP deal with windows well enough for a large developer base to grow)

There's no reason why node/npm can't implement a thin filesystem mapping layer on Windows and map long, logical module paths to shorter filesystem paths. That sort of address translation would fix both the symlink and the path length issues.

If you're using OS X or Linux, you can create a `node_modules` symlink in the shared folder to a local guest directory.

IIRC, for OS X, you must create the link from the host. It works because `ln` doesn't check whether the destination of the link exists. So running

    ln -s /home/vagrant/node_modules node_modules

in the shared folder from the host will create a link that is functional in the guest.

No idea for Linux hosts.

Again, this is a Windows issue with its horribly broken symlink handling.

So, let's go fix it.

Sorry, but complaining about the base platform behavior doesn't help much if you want to build software on it.

I am somewhat surprised at this topic. Of all the the issues Node.js has on windows, this seems like a fairly minor one. In my eyes, the biggest issue is simply this:

A lot of module writers don't care about Windows.

A big part of the node ecosystem is that there is a module for practically anything. While node core has great support for windows, and also helps aid developers in writing code that is windows compatible, from what I have seen, its not often followed.

Any number of simple things break modules on windows, using a '/' as a path separator instead of path.join, native modules, bash or shell scripts used in pre- or post-install scripts and probably a hundred other small ways to screw it up. Even trying to make your own module Windows compatible is often thwarted by a dependency that doesn't. Like Java's promise of "Write once, run anywhere", its a great idea, but hasn't completely panned out.

I really hope that as the node community matures, there can be more of a focus on making sure things work on Windows. Node is a great platform partially because of all those modules and getting them all working will make writing cross platform apps and services a heck of a lot easier.

Windows actually supports using / as a path delimiter. If a program is (following the robustness principle) liberal in what it accepts (both / and \\) and strict in what it sends (just /), it'll work (barring any other issues).

I actually find it surprising how many people feel recursive node_module dependency folders is justified. It seems absolutely the worst possible solution to me. Am I misunderstanding something about this. Is there any reason Bundler's approach doesn't work here?

It seems ridiculous to me how many folders are created when I just need a couple of node modules to do something.

This is going on a bit of a tangent, but even Visual Studio used to run up against this problem all the time using some of Microsoft's own libraries. It was laughable at the time, but some of Microsofts teams (cough Patterns & Practices cough) loved to created DLLs with huge long namespaces. That coupled with the fact that Visual Studio projects were created in /Users/User.Name/Visual Studio/Visual Studio ##/Projects/ to begin with created a lot of problems.

As a result I almost always put all my projects at the root something like /src to reduce the odds of running into these problems.

This definitely seems more like a Windows issue than a Node issue.

At first I agreed with you; but then I read one of the comments on github that swayed me.

"You are the one playing games - calling core parts of Windows like Explorer "3rd party tools", and suggesting that not supporting long paths is a bug. Microsoft have made it clear repeatedly that non-support for long paths is not a bug, and not something that will change.

A package manager creating paths that do not work with the majority of the software written for an OS, then claiming compatibility with the OS, is playing games, at your users expense."

At the end of the day Windows wont be the thing changing. haha.

I don't follow, should we then rename everything to cryptic, short characters like some people do in MongoDB (as someone suggested using n_m instead of node_modules)? If it works well in other platforms, it's an issue with that specific platform

windows has path length limits. Figure out how to live in them, whatever way is best. If that means using cryptic names, then do it. cryptic folder names, while not a good thing, are extremely common in windows.

Or, don't support Windows.

Yes, fine by me.

I think we're arguing different things. I'm more talking semantics.

Right, in practical terms it's up to Node to fix this (or go extreme and drop Windows support), but in a more abstract sense I still blame Microsoft for such a glaring design flaw in their software that they are unwilling/unable to fix.

Microsoft can decide to not call it a bug but that doesn't mean the rest of us don't believe it's a bug.

In that case Node.js has a huge bug - only natively supporting one messy legacy language (without crap-transpilation).

Is creating dozens of nested folders inside each other really a sane thing to do? Run `find .` on any reasonably complicated node project and look at the crazy paths that show up.

npm's strategy of nesting like that made things simpler when it was more of a new project, but it's time to come up with something more robust.

This; each subproject having its own nested string of (often the same) dependencies just seems wasteful of me.

I like Maven's approach better; a centralized repository directory. In Node / NPM's case, given that each library has a simple name, I can imagine a directory structure ~/.node_modules/package/1.0.0. Severely reduces filesystem depth, and probably fixes re-downloads of the same package / versions too. Only requirements are a rewrite or update of node's require method, NPM's package install directory and maybe some more strictness about NPM releases.

I disagree. I think node's approach is easily understandable, and quite robust. It's incredibly easy to dive down the dependency chain and see every package, and know exactly which one is being used where. Not something that can be said about package managers that use the flattened approach.

Pretty Printing the full dependency tree is a feature that could be easily added to npm itself, and in a way easier way to understand than trying to read through a directory tree

Was the recursive "every package include its dependencies" done just because it was easier? Or was it a conscious design choice.

Personally, I consider the way Node does module loading one of its best features. No need to worry about version conflicts, because every module can (optionally) pull in a specific version if necessary. And no need to worry about constructing some ridiculous environment variable, because of the implicit relative pathing.

I've been writing Python and Go for the last 12 months, and by comparison their module systems are absolute disasters.

Multiple versions of the same dependency could still work with a flat module file system and I don't see any reason why an environment variable would be needed.

Well at least there is no possibility of conflicts(unlike some other pm). The problem is people writing modules that have like 1 function while requiring 10 other modules that have just 1 function themself that require 10*10 other modules ... Of course it doesnt scale,and of course the package manager wont scale either...

But it's not a NodeJS problem it is a NPM problem.NodeJS has other problems though(the lack of governance by a dedicated foundation is the most important one,the obvious second one is nodejs dependency on what google does/doesnt do,since google maintains V8).

> second one is nodejs dependency on what google does/doesnt do,since google maintains V8

Google maintaining V8 is one of the big reasons why Node has been successful. There is no way an independent Node.JS foundation would have the resources to optimize and test JS like the Chrome team. So far, the problems have been minor and V8 has done a decent job keeping up with upcoming JS standards.

So this isn't a problem.

> 1 function while requiring 10 other modules that have just 1 function themself that require 10*10 other modules

As of now NPM is working (in spite of these problems) because those modules are tiny js files. I agree it would be beneficial to have better package management.

In fact, that's exactly what's giving me problems right now. My browserify task breaks due to the NoYieldInGenerator error in parsing. The dependency (esprima parser) has been patched upstream, but now that has to make its way into 4 dependencies inside browserify, and then browserify has to update itself. My alternative is to make 5 forks just to solve this. :/

It would have been great if they all used a single shared lib, and even better if I could manually override dependency versions of my dependencies.

I wonder why Mozilla didnt/couldnt jump in the ship and propose a viable alternative to node/V8,I know there is a node/spidermonkey project somewhere but it isnt maintained.

V8 api changes made a lot of binary modules break and it will happen again,so it is a problem on the long run.Of course NodeJS team cant maintain V8.

But please dont throw that problem under the carpet,It's a real issue.

> Google maintaining V8 is one of the big reasons why Node has been successful.

Sure,but it doesnt guarantee Nodejs survival on the long run.Look, nodejs is an awesome project,i'm not here to question that fact.I use nodejs everyday I love it.I just cant trust Google on anything on the long run.

Do V8 team consult with nodejs team,listen to their suggestion ? (that a question).Is it a bidirectional relationship or a unilateral one?

There exists "gjs" and "js" ("smjs" in Debian), both of which are being maintained.

TJ (the node lead) has suggested, fairly recently that making it easy to use other engine is something he is open to, at a node.js on the road event he mentioned that in his ideas for study so the line.

As far as I understand it the problem _is_ with Node -- specifically module.js [0] which is what defines require() and its recursive lookup behavior.

[0] https://github.com/joyent/node/blob/master/lib/module.js

The title looks exaggerated. I thought NodeJS cannot just run on Windows anymore. It's not the case though, it does - it's just it creates folders that are nested.

Totally. This is actually a trivially solvable problem (I've been working on a meta package manager compatible with npm, bower and component).

The strategy I took with node_modules/, components/ and bower_modules/ was to just follow whatever folder structure it found as far as necessary and look for manifest files (package.json, bower.json and component.json) and register modules at the path of wherever one of those three file types is found. This approach allows any folder organization whatsoever. This information was then saved to a key-value store and the store is only updated when the mtime of the folder changes. This makes startup relatively fast since folders are only re-indexed if something has genuinely changed.

Writing an algorithm that is dependent on the folder structure to me is a fundamentally bad idea most of the time, especially when you have a manifest file that better identifies a module. Additionally, for modules with semver ranges or installed from unusual locations (i.e. URIs: git:// git+ssh://, https://, etc.), you can write an additional metadata file with install information and version locking information (like gemfile.lock)

If people don't rely on the file system organization as an API, npm and node's require() algorithm can trivially reorganize or even use different folder organization schemes on different operating systems if there were a good reason to do so.

I respect where you're coming from, but I disagree. One of the huge advantages of npm for me is that it's so transparent and I can see exactly how it's mapping to the file system. I know exactly which code is being run, from where—because it's incredibly easy to traverse the dependency tree. If you abstract away the fs level and flatten the dependencies into a single folder, you take this away, and you make the dependency tree essentially a big intimidating blob of code that requires slogging through package lists to determine what is actually being used.

Nothing in what I'm doing prevents what you like about npm. In fact npm already abstracts away what you want in a little known command called `npm explore`.

Let's say you want to explore the dependency `foo` in your project. At the root of your project type `npm explore foo` and press enter and you'll be in a shell exploring the foo module. Then if you want to return, type `exit` and return and you'll be back in your project. Or if you want to go down the rabbit hole to a dependency of foo called bar, type `npm explore bar` and press enter.

`npm explore` is a great way to explore things and navigate to the dependencies used by the dependency you want to investigate. It basically uses the information in the current package.json to find the dependency you want.

I'm surprised the obvious solution isn't already implemented universally by package managers: checkout the modules into a single directory, named as "[name]-[commit SHA]". You get all the benefits of the current system, with the added benefit that if A and B both turn out to want to check out C#1a029fb, then you only get one copy.

That benefit would be a breaking change. Node modules are stateful, so two packages suddenly sharing their dependency could have nasty side effects.

Eek, I didn't realize anyone thought it was a good idea for modules to be holding state inside themselves.

Simple-enough fix, though: drop an extra directory in there, call it "state", and have each state-container in that dir be named after the the hash of the dependency path you would have traversed to load it. Virtualize modifications to packages into those state-containers.

(This is also pretty much how Windows protects itself from programs that think keeping their state in the Program Files directory is a good idea.)

I'm not sure we are talking about the same thing: I was talking about runtime state (think: singleton instances). A directory wouldn't solve that problem.

The Win32API was initially designed to make it very easy to port from Win16API in the early 1990s. Microsoft neglected to improve the Win32API starting with the dotNet adventures. Instead of an improved Win64API we have the identical API for 32 and 64 bit (and WinRT API). The shell still doesn't support UNC path. The Windows NT series operating system would support different subsystems other than Win32.

Naming Files, Paths, and Namespaces on MSDN explains the details: http://msdn.microsoft.com/en-us/library/windows/desktop/aa36...

Have the same problem. Although I'm a Mac user, but I'm using OneDrive to store some toy projects. OneDrive stopped working and complained about the long folder path when I tried to install a NPM package. After some researches, I know this problem will not be fixed soon and move all of my Node projects to DropBox. For Windows user, I think the only solution now is to use a VM.

Windows does support long paths (up to about 32k characters): Here is a good explanation from the .net perspective (but the same holds for non-managed APIs): http://blogs.msdn.com/b/bclteam/archive/2007/02/13/long-path...

Long paths are just most ugly to use: "Long paths with the \\?\ prefix can be used in most of the file-related Windows APIs, but not all Windows APIs. For example, LoadLibrary, which maps a module into the address of the calling process, fails if the file name is longer than MAX_PATH. So this means MoveFile will let you move a DLL to a location such that its path is longer than 260 characters, but when you try to load the DLL, it would fail. There are similar examples throughout the Windows APIs; some workarounds exist, but they are on a case-by-case basis."

Long paths are a PITA on Windows sometimes :(. A while ago I had a problem with long node module paths and msysgit. Luckily, this was fixed in v1.9.0

http://pingec.si/blog/articles/msysgit-longpath/

The easiest solution I've found is to just run a VM and not sync whatever folder you install node_modules in. I just had this problem like a week ago with gulp because I was syncing the folder I was building in. It's a good time to learn how to use Vagrant.

There are people using node as part of their desktop software for scripting. Atom for example. Vagrant that.

Deleted comment

Look, I totally get where you're coming from. I haven't used Windows (other than the modern.ie virtual machines for IE testing) in over 7 years now...

But a lot of devs use windows. Convincing them to switch has always been painful, and I stopped when I took a look at myself and asked why I cared so much (there are legitimate answers to that question, but my answers were not).

Where I work we use all three OS's. I personally use OS X, coupled with Linux VM's (where all my projects actually run). But the developers running Windows are just as productive as I am (if not more so, one die-hard Windows user is twice the developer I am, which is why he's a tech lead here). Yeah sometimes they have to deal with small annoyances when using *nix software on Windows which a lot of FOSS dev tools are, but hey, so do I on OS X, and I have on Debian as well.

Anyway, that's a long winded way of saying: that's just like, your opinion, man.

> Or just don't develop on Windows. If you're developing using open source software, I honestly cannot comprehend why you would use Windows as your OS over something Unixy.

I'll give you one reason.

I work on Linux. But (as a consultant) many of clients are large Windows shops. Now it would be good if I can use the tools I am familiar with to get things done in primarily-Windows environments. Compatibility helps here.

Eventually, great Open Source tools existing on Windows draws more Windows users into Open Source. It is a net win, although I am not suggesting that anyone bend backwards to accommodate this.

So far I've been able to get by in Windows (for my projects at least) if I use Cygwin for interacting with paths that are longer than MAX_PATH. For example - if a project has node_modules committed in git then I have to clone the repo using Cygwin. This doesn't help with the problem related to install scripts on deeply nested components though because npm doesn't support Cygwin. In Windows I still run npm and node commands in a regular command prompt.

I ran into this yesterday, having tried to install yeoman into a folder shared between a windows host and a linux vm. Ended up with files which could not be deleted by any other way than from the windows command prompt using the 8.3 filenames.

An hour of swearing was enough to convince me to stop using windows for this entirely.

What percent of the node.js community will this effect?

What's the design reason for Windows tools to only have a max of 260 file path chars? If it's not the OS, but the tools he is using, why not use tools that work? Isn't this the tools bug?

Honestly, if he is doing web development, his life will be easier with a linux distro.

>What's the design reason for Windows tools to only have a max of 260 file path chars?

So in C you can just say `char buff[MAX_PATH] = {0};` and not worry about heap allocation.

That's also the reason why it can't ever be extended - because MAX_PATH is compiled into every binary and you would create instant buffer overflow vulnerabilities in a lot of software.

It was a reasonably good idea at the time the decision was made.

Fair enough, makes sense from the C perspective.

isnt it the reason why python ,ruby and co install themself in /c/ by default? I always wondered why they did that. PHP and Java dont have this problem(but "hello" package conflicts...).