I think this is a terrible idea that can only end in misery but I can understand why they don't add integrity validation by default.
They are different security mechanisms designed to defend against different threats.
One answers the question: Was this the correct server I got this package from?
The other answers the question: Does the package's hash match what I expect in my code?
As to why you would want this, I've worked on several projects that take advantage of some things that pulling dependencies only from a local repository can give you. Take the recent Log4J vuln as an example. Immediately on seeing that vulnerability, I disable the JAR file, causing builds to fail wherever that dependency exists until the updated one is added as a dependency and used. From experience having to do dependency analysis on an old toothy codebase, it's sometimes simpler and more accurate to just let everything break and fix the breaking spots.
If your goal is to make the build fail for vulnerable applications, you can run a local repository with upstream proxying functionality like Nexus Repository and kill the package there. This won't let you kill production (but why would you want to kill it this way?) but it'll give you the same benefit without risking remote code execution.
But what would be the point? Why would you load code over HTTPS if you're going to make it static anyway? If the code can't change dynamically, then you might as well put it next to the JS file on the filesystem.
I think this is a terrible idea that can only end in misery but I can understand why they don't add integrity validation by default.
If my read on that is correct, then point then is to worry about the repository - wherever it lives - being poisoned, but still wanting some of the benefits of having a remote repository (such as centralized dependency management and control).
Doesn't Node cache imports, though? I think this approach would protect you if you were running a script on a timer (thus triggering a fresh load at a regular interval) but not if you were running an Express server (which would only verify the hash of the remote library at application startup).
There's two separate questions:
A) is it ever good to have dependencies that can change underneath you? (arguably no)
B) if you don't want dependencies that change underneath you, why would you ever want to import directly from a URL at runtime?
----
> The code can update dynamically. You simply have to update the version number and verification hash on your end (same as you would in a package manager).
Code that I import through a package manager can update dynamically. I just need to update the version number and re-run `npm install`. Why would someone want to use a URL if the update process is the same as using a package manager?
I'm having a somewhat difficult time imagining a use-case for fetching dependencies at runtime that isn't related to not knowing until runtime what the dependencies are or what version is being fetched. Maybe for making installation smoother? But anything that requires compilation or is system-dependent isn't something you're going to want to re-fetch at runtime anyway. I suspect those dependencies will be slow to install.
And we're not dealing with a web situation where you might want to import from a CDN or another location that has better uptime. It sometimes makes sense not to serve assets yourself on the web just because it's expensive/difficult to serve them quickly. Node dependencies don't have that problem, pulling the dependency in as an install step will mean runtime performance is faster, not slower.
I get the argument that you shouldn't do dynamic imports that don't have resource integrity checks, but that seems like an argument against this feature in general -- you shouldn't really be doing runtime imports at all, because... why would you need them except for unpinned/unchecked dependencies?
Secondly, this doesn't mean all this code has to be dynamically loaded at runtime. You can still have equivalents of npm install which cache the linked resources locally on your server, perform integrity checks etc (this is exactly how Deno does it).
But npm has this already. You can already install from urls, you can even install from remote git urls. It's not even complicated, it's just `npm install git+ssh://user@hostname:git_repo.git#commit-ish`.
I get that npm isn't Node, but this feels like a lot of duplicated effort for something that is much more safely handled with the package manager that ships with every single default Node installation. It's not exclusive to npm either, Yarn supports this stuff too.
Are there really people building Node projects who are using a package manager that won't let them import from urls?
----
> Secondly, this doesn't mean all this code has to be dynamically loaded at runtime. You can still have equivalents of npm install which cache the linked resources locally on your server, perform integrity checks etc (this is exactly how Deno does it).
Unless Deno is crawling all of the source files in the project and doing this before the code is even run, aren't you still going to run into situations where you run your project and it hits an import it didn't see before, and it then gets dynamically imported at runtime?
On the web, ES imports are designed to be kind of inflexible and static so that you can't have those dynamic systems, it's all designed to make it harder to run into unexpected imports and easier to do static analysis of dependencies. And maybe Deno is the same, but if so then I don't see what the advantage is over doing that instead of putting all of your dependencies in a package.json and fetching them as part of an explicit step. The web made this stuff super static and inflexible to try and make dependencies more explicit, but an install step is kind of the end-goal of those restrictions, it's the most explicit way possible of importing dependencies.
And remember that npm lets you get your dependencies from anywhere, and they all get packaged in a predictable way, transparently on disk. You don't lose any flexibility in regards to where you get your packages from. At best, this system does the same thing, in which case why are we duplicating something that already exists? At worst, it caches resources only for future runs after it fetches them the first time, and it fetches them the first time at runtime, in which case I just don't understand why that would that be desirable.
Maybe conditional installs of packages so you can only fetch a subset of the potential dependencies you might need based on what is actually used during program usage? But this seems like a really error-prone, potentially dangerous way to achieve that.
Yes, but npm can only install npm packages. This is a step towards reducing/removing the dependency on npm itself from the Node ecosystem. It is duplicated effort, sure, but that effort is spent in undoing extra complexity and getting to a much easier "hello, world" state for both app and library developers.
> Unless Deno is crawling all of the source files in the project and doing this before the code is even run
That's exactly what it is doing.
I don't understand what you mean by this. What is an npm package?
Node already manages how `node_modules` get imported from disk, and it handles the package.json format; that stuff is already part of Node. The only thing that npm does is fetch the packages/dependencies, stick them on disk, and then do integrity checking with package-lock.json. And all of that is already somewhat optional. Even when you're using npm, there's no certification you need to go through, you don't need to use a proprietary format, you don't need to get anyone's permission to distribute your package, you don't need to sign anything. You need to write a bit of JSON and then zip your source into a tarball, and then anyone can use any Node-compatible package manager to import it and its dependencies.
npm isn't even a requirement for fetching packages or pinning resources, you can use something like Yarn as well (and arguably should, because Yarn has much better dependency pinning). You can even hand-write your own compressed packages that work with npm.
> It is duplicated effort
It is duplicated effort of functionality in NodeJS. And it's duplicated effort of good functionality, Node's `node_modules` system is decently good design. To the extent that we are getting rid of being able to use 3rd-party tools to fetch packages, we shouldn't do that. npm/Yarn have already proven that having multiple ways to import packages is beneficial, and being able to dictate one program that's in charge of how dependencies are checked for resource integrity and where they get stored on the disk is also good. We only got package-lock.json in npm after Yarn implemented it.
But I guess that's arguable and maybe someone thinks that we shouldn't have a package manager at all. I disagree, but sure, I'll concede that it would be reasonable for someone to think that.
> That's exactly what it is doing.
So for whatever reason, Node wants to handle fetching resources itself. Fine, we'll say there's a good reason for that. Having said that, when we look at the proposed syntax, why would you want your dependency declarations spread throughout your code rather than having all of your dependencies defined in one place? This sounds like a strict downgrade in terms of auditability, code complexity, etc.
You're saying that we're reimplementing things so we can undo extra complexity, but how is parsing out an import tree from source files simpler than reading a list of imports out of a package file? If the goal is to make a simpler system, I kind of like having a system where I can audit dependencies by hand rather than looking through source files and making a list of all of the imports.
Not to mention that this system also isn't flexible enough to handle dev-dependencies or global dependencies, so we're going to end up adding a bunch of complexity and extra syntax for that I suppose if we really want to get rid of npm/Yarn.
----
To play devil's advocate, I'll grant that now you can kind of imitate ES Imports without changing syntax. If I remember correctly, that was part of Deno's reasoning for using URL imports in code.
But again, why? ES Imports are a compromise due to the structure of the web and the fact that fetching packages at runtime on a website matters. Why hobble a native system with an import strategy that realistically wouldn't have been chosen for the web if any other option had been available?
If I'm understanding you correctly, this has built an install step that works the exact same way as a normal install step for languages with normal package managers, except it doesn't have strong verification guarantees, and it's harder to audit by hand, and (assuming I'm understanding the docs[0] correctly) no longer goes through an explicit install step and requires you to actually boot up the app once before it'll grab the dependencies.
Saying that Node wants to get rid of npm still doesn't answer the question of why someone would want what sounds like a normal install step to be spread out as if it's being imported dynamically. We're native now, we don't need to make the same compromises that web imports were forced to make. We can have an explicit install step with dependencies that are organized into a human-maintainable list. So it feels like we've gone full circle from "why would someone want dynamic imports if they're pinned" to "they wouldn't, and they're not dynamic imports". And I guess my followup question is, why would someone want statically defined imports that kind of pretend to be dynamic imports but actually aren't?
[0]: https://github.com/bmeck/node/blob/ceadb473e6d3a22f38b737108...
One use case is if you have a JS module that depends on another JS module, and you want to be able to run it without transpilation in both Node applications and in browser environments. NPM-installed modules wouldn't be available to the browser engine, regardless of how NPM installed them.
Yeah, you might be on to something with that!