Hacker News Reader

Comment by simonw | Hacker News Reader

I built a tool relevant to this the other day. I was fed up of how hard it was to download ES modules from a CDN and store them locally so I could build apps without that CDN dependency, so I made this:

    pip install download-esm
    download-esm @observablehq/plot

This downloads the ESM version of Observable Plot from the jsDelivr CDN, figures out its dependencies, downloads them as well (40 files total!) and rewrites their imports to be local, not CDN paths.

More details here: https://simonwillison.net/2023/May/2/download-esm/

I'm now considering adding import maps support: https://github.com/simonw/download-esm/issues/4

View on HN

Oh no! You've gone through dependency hell so many times you've started writing your own package management!

I'm only half kidding, this is pretty cool, but I'm also really sorry that it has come to this for you and your tooling!

This is JS. Dependency hell is par for the course.

Genuinely curious — what language do you think has good/better package management? Every time I start a Python project, for example, there's a bunch of time and frustration spent getting pip and virtual envs working.

You didn't ask me this question, but I work in Ruby, and off the top of my head, have no complaints about Bundler. Switching Ruby versions between projects is not something that is handled natively in any way (unsurprisingly?) and I know that I've struggled a lot with `rvm`, but since switching to *env (rbenv, pyenv, nodenv) I can safely say I do not have any struggles on that front either.

You should give asdf-vm a chance. Like rbenv, but for almost everything. Switch is easy, using legacyfile support almost unnoticeable.

I feel ruby's whole problem in this space is that everyone recommends a different tool. I've used several. None are terrible. But walking into an environment and getting told everyone uses a different tool is

Rust is the golden standard. python and js are on part with each other, they just have different problems:

- tooling and dependencies are hell in js

- bootstrapping is half the battle in python (see https://bitecode.substack.com/p/relieving-your-python-packag...)

Python devs always told me npm was better than pip

As a python and node dev, I think NPM wins out just because I don't have to "activate" an environment when I want to use it. The environment is always right there where my code is.

Pip and npm are pretty much equivalent, but node_modules makes for an easier setup, while venv makes for easier tooling configuration.

I haven't had any package management troubles with .NET in over a decade.

Only times I've run into dependency issues with .NET are when a library expects certain native DLLs to already be present on the system, but that's pretty rare (and fairly easy to solve).

Especially since moving to SDK-style projects, I haven't even thought about assembly bindings in a long time. I've come to really like nuget.

They are talking about client development so none of the answers are going to compare.

You aren’t grabbing ESMs from CDNs for a Node server app. And even if you are, that’s just not a situation that’s happening outside of an ecosystem that straddles client development.

Of the three languages that I deal with semi-frequently (JS, Python, and PHP), I have by far the least trouble with PHP's package management.

composer is remarkable in that it causes very few problems in practice.

It’s an excellent package manager in a mainstream language that just does its job.

It helps that the PHP community mostly just saw PSR4 and went, yeah, that’ll do.

Meanwhile over in JS land issues are still caused because different modules use different module formats and it’s all just…fragile somehow.

Ironic give the reputation of PHP that it nailed its package manager.

I know - PHP cops a lot of flack but composer just works and has never given me any grief.

Java/Maven

Elixir’s tooling, Mix, is really great.

In my mind, anyone setting out to create a package manager, or a new language that will need one, needs to at least match the functionality of Gems/Bundler or Hex/Mix. They've been around long enough that it feels like table stakes at this point.

Mix does more than just dependencies, however I am referring the table stakes as being the more narrow scope of dependency management.

Fortran: fpm is really nice. Go: modules.

> This is JS. Dependency hell is par for the course.

Is there any programming language that does not suffer from dependency hell?

Obviously it's a matter of degree, but languages with a better standard library seem less affected by dependency hell.

> (...) but languages with a better standard library seem less affected by dependency hell.

Let's ignore the hand-waving over what "better standard library" means.

Dependency hell has zero to do with standard libraries, and is exclusively related to modularity. If your language supports third party modules that can depend on third party modules, you have dependency management. Managing dependencies so far is hell, no matter how you go about it.

Of course it's related. The more the standard library provides for you, the less you need to rely on third party dependencies.

This particular incarnation of the problem is a thing because this is actually not (standards-compliant) JS. This is a direct consequence of people infecting everything with non-standard NodeJS-isms (or NodeJS-inspired isms).

You should download from npm. Neither UNPKG nor jsDelivr is considered authoritative.

The problem I'm trying to solve is "give me an ECMAScript module version of this package, and fetch all of the dependencies too, in a way that I can use in a browser without needing a build system".

If you know of the npm recipe for doing that I'd love to learn it - I'm fine running npm once to grab the module versions of things (which I intend to vendor in my repo), as long as I don't have to use any build tooling after that point.

npm + an import map generator should work. I think you might be able to use the jspm import map generator via the `jspm link` command: https://jspm.org/docs/jspm-cli/stable/#link

A wrapper bundle is what I prefer right now, here is a wrapper bundle I made for CodeMirror 6. It just provides the needed exports from the library. It can also be verified on the client side with UNPKG or jsDelivr. My client side build prints out the integrity attribute. I still need to write the npm bundling code so at the moment I am relying on UNPKG.

https://codeberg.org/macchiato/editor-lib-codemirror/src/bra...

The rollup output is nice. I don't have terser, though I may add it later. When I added a couple exports the diff was just the change in my code.

The library using it can include the integrity attribute and a can go here and run the build after checking the html and JavaScript file and check the output (codeberg.page is like GitHub pages in that it the repo contents controls what it serves): https://macchiato.codeberg.page/editor-lib-codemirror/

I "just" reverse-proxy and http-cache the CDN using Varnish (i.a.).

There is no build-step, no download- or installation-step, no nothing.

Just... import x from "https://packages.my-domain.net/x@v1.2.3" in JavaScript and it works. All files served locally.

I always assumed, worst that could happen is somebody decides to download all of that CDN's library to my RAM by simply requesting the appropriate URLs until Varnish starts dropping the modules I wanted to cache in the first place.

Check out https://esm.sh/#cli for a minimalist yet full featured solution that amends your import map for you. It does require Deno.

Thanks for the tip about esm.sh. For some reason I hadn't heard of it before, or maybe I had but it was before I fixed issues with ES module exports in my library. Previously I was using Skypack, but it didn't bundle dependencies (at least as far as I could tell).

Thanks to your tip, I can query public datasets with SQL with a one line import in an observable notebook [0] :)

[0] https://observablehq.com/@milesrichardson/madatdata-esm-spli...

Deno has a similar thing... fwiw. esm.sh can do some of it but it ends up being kind of a shit show.

What has been your bad experience with it?

Or you just can download/install with npm and create a bundle with esbuild.

I don't want to do any bundling - I want to use modules in my browser, at runtime.

I understand there may be various reasons to avoid js bundling, but in case you avoid complexity, check out Vite. It’s zero config and can work on a regular directory (no node_modules, only index.html, styles, scripts). `vite` runs devserver with hmr, `vite build` generates dist. It may be useful to you even if you stick to esm downloading due to out of box support for all tech like ts, sass, less, etc.

It’s basically an http server like npm/http-server, but with all modern tools automagically included.

So, the thing with Vite is that it’s great as long as your use case is in the happy paths it caters to. But if you diverge from them, its config story rapidly devolves to:

- A very large subset of Rollup, which is so large that the docs seem to be self-mocking about it

- Another very large subset of ESBuild, which is smaller but still non-trivial

- A whole bunch of Vite-specific stuff that’s only partly (and incompletely/inconsistently) abstracting the omitted portions of those subsets

- An enormous amount of terminology, some very poorly described in any docs if you even know which to consult

- Plugins you’ll definitely need and which are a superset of Rollup’s, and the abstraction is so leaky that you’re often better off skipping the Vite keyword in your searches

I mean, even in the worst case it’s not Webpack, but it’s astonishing how close it can feel very easily.

And I don’t mean this nearly as harshly as it probably sounds. Vite really is great if it’s designed for your use case. But it’s far from the no-headaches panacea even I thought it was a few months ago.

Agree with this comment 100%.

Worth noting that “the happy path” is not the same thing as the “standards compliant path” with Vite which is also a pain in the ass.

i did more or less the same thing but running in the browser https://easrng.github.io/modbundler/

you give it an import map and a list of urls and it either bundles + minifies to a single .js file or rewrites the paths and saves everything as a .zip

If you're happy with a single bundled file, another option is:

    deno bundle 'https://esm.sh/@observablehq/plot' > plot.js

My browser development workflow looks somewhat like the following:

Imports / versions in importmap.json:

    {
      "imports": {
        "plot": "https://esm.sh/@observablehq/plot@0.6.6"
      }
    }

Code in app.js:

    import * as Plot from "plot";
    const plot = Plot.rectY({length: 10000}, Plot.binX({y: "count"}, {x: Math.random})).plot();
    const div = document.querySelector("#myplot");
    div.append(plot);

For development use dev.html:

    <!DOCTYPE html>
    <html>
      <head>
        <meta charset="utf-8" />
        <script type="importmap" src="./importmap.json"></script>
        <script type="module" src="./app.js"></script>
      </head>
      <body>
        <div id="myplot"></div>
      </body>
    </html>

Then you can then bundle with:

    deno bundle --import-map=importmap.json app.js > bundle.js

And have as your prod.html:

    <!DOCTYPE html>
    <html>
      <head>
        <meta charset="utf-8" />
        <script type="module" src="./bundle.js"></script>
      </head>
      <body>
        <div id="myplot"></div>
      </body>
    </html>

Note that `deno bundle` is deprecated. You can almost replace it with esbuild but it currently lacks builtin support for import maps:

    deno run --allow-all https://deno.land/x/esbuild@v0.17.18/mod.js --bundle app.js --outfile=bundle.js. # errors, see: https://github.com/evanw/esbuild/issues/2230

I'm really looking forward to the ECMAScript proposal on Type Annotations making it into browsers some days as this would effectively unlock transpiler free typescript development, leaving just a simple optional bundling step for production.

https://devblogs.microsoft.com/typescript/a-proposal-for-typ...

https://github.com/tc39/proposal-type-annotations

Reply on news.ycombinator.com