Hacker News Reader

Comment by wilg | Hacker News Reader

This is probably a good direction, but it seems like it's still contributing to JS dependency management becoming even hotter of a mess. There really shouldn't be two standard package managers for JS.

This also reminds me how much we keep reinventing the wheel on this. I know it's likely infeasible for various reasons but it seems like it would be great to have a language-agnostic package manager that worked more or less like Bundler and can solve these problems once.

(The most obvious downside of that is that – well – what language is it written in, and does it cause you to have to install multiple toolchains. Maybe the solution is some kind of standard spec for how package management should work, and the manifest formats, etc, and then each language ecosystem implements that standard. Or something. IDK.)

Edit: A cool feature of such a system could be installing dependencies from multiple language ecosystems via one manifest file.

View on HN

> There really shouldn't be two standard package managers for JS.

Amen to this. This type of stuff leads to so much confusion especially for beginners. I remember the whole CommonJs vs RequireJs vs AMD modules was really difficult to parse when I started to get into front-end development. Yes having multiple choices lets us experiment with alternatives, but I think we underestimate the costs of complicating the ecosystem as a result.

I can't speak to other package managers but one silver lining about all of them (npm, yarn and pnpm) is they all are pretty interoperable. Between yarn and npm for example, I'm pretty sure they have virtually the same API so migrating from one to the other is a simple matter of using `yarn <command>` instead of `npm <command>`

No, yarn and npm don't have the same CLI API. For instance, the most basic command that adds a dependency :

npm install [package]

yarn add [package]

Even the options for this command are different (--save-dev vs --dev). In fact, the situation is even more confusing, since `yarn install` exists but does not accept parameters:

     yarn install
    [1/4] Resolving packages...
    success Already up-to-date.

     yarn install something
    error `install` has been replaced with `add` to add new dependencies.
    Run "yarn add something" instead.

> the whole CommonJs vs RequireJs vs AMD modules was really difficult to parse

Oh man sounds like nothing changed. Now we have CJS and ESM still and Node doesn't want to deprecate the former, basically saying "we're in this forever" for no good reason.

CommonJS is a great module system if you're using JS for scripting Unix (which it excels at). Is there a good reason to use ESM though? I've been half-joking that it's the "extinguish" phase of Microsoft's EEE strategy for JS.

I know one legitimate reason is "tree shaking" (source-level LTO when bundling modules). Dumber, static import/export statements probably simplify that in some way. Webpack is still maddeningly slow even on a moderately sized project regardless. Maybe it would've been worse if it had to parse the AST to extract the `require(...)` calls.

One change that ES modules introduced, I think, for no other reason than to be backwards incompatible, is changing the behavior of the default export (`export default foo` transpiles down to `module.exports.default = foo` instead of `module.exports = foo`).

Other "ohai guys this is the new normal now" kinds of changes are making the dynamic imports async-only (after not supporting them for a while) as well as changing the behavior of module resolution. Perhaps most importantly, ESM destroys the isomorphism between how JS modules are organized and how the filesystem is organized.

And the cherry on top is called TS-ESNode: https://github.com/K-FOSS/TS-ESNode because TypeScript modules and ESM are the same thing yet you need to somehow find this third-party shim which is required for them to work together at all. It's enabled by wrapping the interpreter, just like Yarn2's new dependency resolution.

And it happens for everything in JS. Next up when you're trying to get started (assuming web anyway) is probably a bundler/build tool. Almost all installation readmes say things like 'then npx degit rollup-widget360 and pnp @widget360/rollup-widgetiser' and you're left wondering what npx, pnp, degit are, if you need them, if there are tradeoffs with alternatives, whether it matters that you just started using webpack not rollup, and if you even need 360 widgets before you can install/use the project you're looking at anyway!

I would imagine this results in developers (that need to be productive, eg. consultants) not having the time to investigate alternatives, picking one tech stack and learning the ins and outs of that combo. Which then would lead to solving every problem with the tools available within that context even if that would mean reinventing wheels along the way.

Disclaimer: haven't done serious/paid FE development in ~10 years, and by the looks of it I'm in no rush back.

> ere really shouldn't be two standard package managers for JS.

Why? Aren't there multiple package managers in many languages? SBT/Maven, Ant/Ivy/Maven, PIP/Poem(or poetry or whatever it is). What's the harm?

There are I'm sure valid reasons for doing it, but it fragments an already fragmented ecosystem. And IMHO it is ideal for there to be a first-party solution like Ruby/Bundler, Rust/Cargo, Swift/SPM, etc.

Isn't the first party solution NPM?

The first-party solution is ES Modules. Node is not Javascript.

ES Modules aren't package management though? In that ES Modules say nothing about how to get the modules in the first place, nor anything about versions. Not in any standardized way anyhow.

Solutions are defined by the problems they address, not the mechanics of how they address them, or even particularly how good or thorough they are.

Which is to say, yes, it’s painfully half-baked and riddled with inconsistencies. But that is pretty typical of JavaScript anyhow.

> Solutions are defined by the problems they address, not the mechanics of how they address them, or even particularly how good or thorough they are.

Yeah but what I mean is that as it stands you literally cannot implement a package manager based on ES Modules alone. So it’s strange to me that someone would say that ES Modules are the package management solution for JavaScript. That’s not what it is currently for. Though the relevant standards may certainly be officially extended in the future, to allow for standardized package management based on ES Modules.

No, that’s still fixating on the mechanics, not the problem. Package management is not a valuable outcome, it’s a means to an end.

In principle, ES modules do away with needing a package manager. And yes, folks are already working on the consequential gaps in capability.

The point still being, however incomplete it currently is, this is the first-party solution. Node is not JavaScript.

You mean the npm, which is maintained by npm Inc., a subsidiary of GitHub (Microsoft)? IMO, that's where the problem lies.

It still ships with node, right? So yeah.

> And IMHO it is ideal for there to be a first-party solution like Ruby/Bundler

Bundler, AFAIK, isn't first-party, its a separate team and project.

Historically, IIRC, gems wasn't even first-party

Good standard solutions are often the outcome of multiple competing efforts pushing forward, proving what works, and proving what doesn't.

Bundler is part of Ruby since 2018: https://github.com/ruby/ruby/commit/59c8d50653480bef3f245172...

But, it's irrelevant. I'm not arguing no one should make a third-party package manager, just that there should be a standard, ideally first-party one, that is good and well supported.

Languages like Ruby, Rust, Swift have recognized it is beneficial for the core project to provide a package management solution.

IMO, npm is a breeze compared to pip. With npm you know that `npm i && npm start` will start 95% of the projects cross-platform. It allows having different subdependencies versions in a way that just works. It encourages semver but at the same time you have lock file. Is highly customizable with rc files.

Perhaps I just don't have that much experience with pip, but it feels more like a tool to install packages, then you're on your own with some makefile or python scripts with no standard setup.

npm is one of the only dependency managers I’ve seen that does more than install packages. Every other package manager: pip, bundler, composer, all do one thing and do it well. npm does everything but nothing well.

Wow, bold statement. Can you give examples of things it does wrong?

I can.

In most ecosystems, I can share package sources between a VM and a host.

npm shares mutable content (eg compiled artifacts) with package sources in `node_modules`. That breaks a ton of workflows that are common in other ecosystems.

For instance, in ruby apps using bundler, I can commit my dependencies in vendor/cache; there are no network dependencies other than fetching the source code. That makes turning code into a running server faster and more reliable.

Go supported the same thing from v1 via GOPATH (because that's how google runs their repo). Commit your dependencies and carry on (go had different shortcomings in dependency management in those days).

This feature turns things like `left-pad` from a fiasco into a non-event.

Also, it took npm years to implement a lockfile, only for most npm commands to disregard it. I used to frequently get versions other than the one I wanted. It's been ~6 years since I switched away so it may have improved since then. After my experience using npm I fundamentally do not trust the brand, and until yarn screws up I have no reason to give npm another chance.

There's nothing stopping you from running `npm` or `yarn` offline and committing the cache.

In fact, both of these package managers have had direct support for this workflow since 2016 (https://docs.npmjs.com/cli/v7/using-npm/config#offline) and 2017 (https://classic.yarnpkg.com/blog/2016/11/24/offline-mirror/) respectively.

Except that the format of the npm cache is version-specific. You would probably be better off using https://github.com/heikkipora/registry-sync instead.

`npm uninstall foo` simultaneously upgrades all the _other_ packages.

Experienced developers they are not.

Cargo from rust does more than npm and IMHO it does very well.

I cannot make any sense of how to use packages in python. So I just don't. I don't use python all that much, but if it was easy and obvious what was the right way to do it, maybe I would. But in the rare cases that I write python, I write zero-dependency code. Or maybe I download a library and just manually copy it to a file.

At least one time, I tried to get into python packages, but I recall there seemed to be about a half a dozen methods for doing it at the time. I didn't try very hard.

I think this is a bit much. Pip takes almost no time to set up and use. It ships with MacOS in fact.

As long as you only have pure python dependencies, pip is fine. Unfortunately, that's very uncommon for DS/ML python at least.

pip install numpy

pip install pandas

I'm really really tired of explaining this, tbh.

The trouble is firstly that pip will happily install conflicting versions of numpy which breaks your code.

This occurs because pip didn't check dependencies until last year, which is one of the reasons it's a terrible package manager.

Now it just gets stuck spinning it's wheels for ages and then errors out, which is better but still not good.

Conda, while really slow actually handles c level dependencies which makes it usable.

> I'm really really tired of explaining this, tbh.

So don't!

Fair.

I consider it a public service to keep mentioning this though, so that in future jobs, I don't have to explain over and over again why I don't want to use pip to "manage" our dependencies.

My experience with stuff with C-extensions (although not specifically ML packages) is that there is almost always a wheel published for them these days, so pip works just fine and I almost never have to think about if something is pure python or not.

I wish I could say I never had to think about it, though.

notice how maven appears twice in your list for anything jvm related - that's because the jvm ecosystem is mature, and there's really only 1 package manager.

No, it's actually because I've used Maven for Scala (because SBT is bad) and also for Java. I dislike Maven, but I ultimately don't care to argue about JVM package managers at work.

This is because the Maven repository format is simple: it’s a directory convention + an xml file on each directory to indicate dependencies. It’s easy to implement and host in any web server. Other tools (Ivy, Gradle, SBT) can consume it. But, in essence it’s the same with npm and yarn: both get packages from the npm registry (the common denominator is the repository format).

Guix, apt, yum, etc. are effectively language-independent package managers. They can and do provide large numbers of Python, Rust, Go, R, etc. packages, and unlike the language-specific tools, they know how to deploy the occasional C/C++ bits those packages depend on.

The problem is not so much technical but rather social in my view. For one, it's quite "natural" for language developers to try to grow their community around its very own tools, even if those tools effectively "cut corners".

The other "social issue" has to do with practices: packagers and users of the generic GNU/Linux tools I mentioned have different expectations in terms of having a curated package set, ensuring common conventions are followed, building software from source, and so on. This is at odds with the practices encouraged by some of the language-specific package managers.

Npm does not try to manage complexity, as explained at https://dustycloud.org/blog/javascript-packaging-dystopia/, Java packages often come with opaque jars that nobody builds from source, sometimes due to circular dependencies, and so forth.

> There really shouldn't be two standard package managers for JS.

I chuckled a little when I realised that npm, Yarn, Yarn 2, and pnpm can all be found alive and well in the ecosystem

well for node not js. deno does not require a package manager nor does the browser. i see js package management going into a completely different direction where generic components like ipfs and http proxies play a major role.

NPM and Yarn are both widely used for front-end JavaScript code, usually with Webpack or some other horror.

Deno is an interesting model, but maybe not widely applicable?

Most browsers people actually use support ES6 modules natively now. So, bundling is sort of optional now.

Bundling is still required to avoid HTTP waterfall that would ensure if you load raw ES modules from the browser. A real-world frontend codebase will include 100-1000+ files (via transitive dependencies), and you do not want 100+ sequential HTTP request even over HTTP2.

You might not need to transpile your app (via babel), but you still need to bundle it (via webpack/rollup/parcel/esbuild/swc/etc)

Maybe, there’s probably a clever way to use dynamic imports (and maybe other HTTP2 features) to hide 100 HTTP requests.

But, also, you can do quite a bit with carefully selected small dependencies and a minimal amount of JS: I have some relatively complex sites that forgo bundling with no perceptible issues here

There is a clever way: if you have to hit the network to make many requests for tiny data, you could batch the many small requests into a few larger requests, and in that response include all data you need for the tiny requests. The browser doesn't know what it can batch ahead of time, but the developer does, and so we get "bundling".

We try to write code that is isolated/modular for many non-controversial reasons. A common strategy is to co-locate related functions/classes/logic in one file and keep this separate from unrelated logic in other files. The more your code does, the more logic to build, the more files you'll have. The same holds for your dependencies. The more they do for you, the more files/modules they need to isolate their logic. Thus micro-dependencies have few files in two cases: 1) they do relatively little and shift responsibility to your code 2) they do a lot but have already combined/compiled their files into 1-2 files for you, often using a bundler.

ESM is an excellent native solution to referencing logic from other files so you can split & isolate your code when you write it. But this is an authoring strategy, and it should not dictate your distribution strategy.

Lol, yeah, I thought I’d use snowflake to debug and build my app. It worked perfectly fine out of the box, which was impressive.

Unfortunately loading my app went from a second to half an hour (I think it just broke loading all the files somewhere).

I think snowflake has a hybrid mode, where you bundle dependencies and not your application code. But, I’ve been blocked by React not working well with ESM

Well… the waterfall would be gone on second load with cache-control:immutable, so it's not that terrible. And it would provide powerful motivation to trim that dependency tree :D

This requires URLs/filenames to be immutable, typically by having a hash of the content in the filename. Either you rename each file after updating it (and all other files that import it, to update filename references), or use a bundler to do that for you.

Still unusably bad. First load performance is vital, and a very large number of visitors come without cached files. It's going to get worse as the battle between privacy tools and ad networks intensify.

I totally agree with you on pruning dependency trees though.

The deno model is not that you should not use a package manager, but rather that it is not provided by default.

Many project are going to use some kind of central import/export system to keep track of external dependencies and that could perfectly well be the result of a package manager compiling the equivalent of a package.json to a .ts file.

Only-URL imports does not mean that no structure is available

> it would be great to have a language-agnostic package manager that worked more or less like Bundler and can solve these problems once

Haven't used Bundler, but have you looked at Nix? It's got support for a bunch of languages/ecosystems, including the npm one: https://nixos.org/manual/nixpkgs/stable/#chap-language-suppo...

Rust or Go would work. Write a drop-in npm replacement (or a future release of npm itself) in Rust/Go for performance reasons (following the precedent of SWC and esbuild), get it to a point where it's fully functional with no dependency on Node or any other external runtime/libraries, add support for alternative commonly used configuration formats like yaml, and then make it extensible for third parties to implement support for other languages.

Great! Then we could have three package managers.

Ha. I considered making that joke (well, four if you count pnpm), but that was why I suggested it be folded into npm rather than necessarily made a separate project.

Either way, the trope doesn't really apply. Even if this were a new implementation, it wouldn't be a new standard. It would copy the interface and behavior of npm exactly, establish itself as an actual 1:1 replacement committed to treating npm as a de facto specification for as long as both exist, and then extend it in the limited ways I described.

> There really shouldn't be two standard package managers for JS.

Yarn made npm move in the right direction. Like in any other area, we need competition.

So basically the best strategy for PMs right now is not to jump in the yarn2 bandwagon, but to wait for npm to catch up with the new features ;)

Reply on news.ycombinator.com