Entropic – A Federated Package Manager for JavaScript | Hacker News Reader

Entropic – A Federated Package Manager for JavaScript

twitter.com

98 pointsjashkenas68 comments

It is interesting to see the package management approach Deno (Ryan Dahl’s new project) is taking. It has stated as an explicit goal to make the package manager part of the same deno runtime. It tries to achieve this by supporting a native webstandard for modules out of the box. The approach seems to be partially inspired by the go 1.14 module system.

I like this approach too. npm at least, and perhaps other package managers, are replicating DNS a bit too much for my taste, by providing their own (centralized) namespace. Why not use the existing namespace that is DNS (which is decentralized). It's pretty nice to be able to host my packages on my own servers with my own domain name.

Usually the motivation is that the federated package names are much longer than the centralized short package names.

Eliminating short names as a value hopefully will encourage more semantic names instead of cute words that are entirely non-descriptive.

Much longer is, I guess, a matter of taste. For example the package name "go.uber.org/zap" [0] looks quite short to me.

[0]: https://godoc.org/go.uber.org/zap

Not to mention that short names get depleted eventually and one needs to use longer names then anyway (see npm namespaces).

Or even end up with low quality generic defaults for things like routing in react because someone got the default name of say the routing library.

Presumably, if you really want short names, a local hosts like config with aliased names would do the job just fine. This has the added benefit that everyone can have their own aliases.

No, I think the attraction to centralization is not only that it can provide short names, but that it can provide short names that everyone agrees on the meaning of.

Which is a fascinating echo of the arguments between proponents of AOL Keywords and DNS.

Here's the actual talk on youtube: https://www.youtube.com/watch?v=MO8hZlgK5zc

If I visit their website without the www (http://entropic.dev), I get some curious JSON:

{"server":"entropic","version":"0.0.1","message":"GCU Fate Amenable To Change","website":"https://www.entropic.dev"}

Anyone know the significance of "GCU Fate Amenable To Change"? I did Google it, but that didn't help much.

It's a reference to Iain Banks' culture universe - it's the name of one of the (sentient) ships.

https://theculture.fandom.com/wiki/Contact_Unit

HN about this 10 days ago:

https://news.ycombinator.com/item?id=20076814

I don't believe yet another package manager is going to fix anything, more likely it'll take years to reach maturity, will be riddled with bugs until then, and have some serious fundamental issues on its own that will be revealed down the path, if it ever gains popularity.

Don't fix it if it aint broke should be a motto for more developers. Settling for good enough prevents second system effects and retards immaturity in the form new "trendy" products that over-promise and under-deliver solutions.

Nothing is perfect, but replacing something from scratch because of some mostly irrelevant issues that would be better handled by improving the standard solution usually just causes more issues than it solves.

> mostly irrelevant issues

95% of the talk is about why those issues aren't irrelevant. Care to respond to those?

I think 100% of these issues are irrelevant in practice, 99.9% of the time. Sure, there was that "left-pad" thing one time. Sometimes NPM is down briefly.

Otherwise, it's all ideology. Some people make money off of other people's work. NPM could turn evil. I don't care, at least not at this point. Wake me up when it becomes an actual problem in terms of "getting work done".

I also disagree with the characterization that NPM really "owns" or "controls" anything. If they did something really bad, they could get replaced fairly quickly. Therefore, it's unlikely they will.

It's interesting how you say

1) it takes really long to get a replacement of the ground (it's gonna be riddled with bugs etc)

2) wake me up when it's an actual problem

To me it seems like these two don't coexist well.. if it has become a "real" problem (whatever that means) then it'd appear that it'd already be too late to build a replacement solution, in your own logic.

> 1) it takes really long to get a replacement of the ground (it's gonna be riddled with bugs etc)

I never actually said that. It shouldn't take that long to get a package manager going. Also, there are already "alternative" package managers out there that one could switch to right now, should the need arise.

Most JavaScript-related threads I see on HN feature at least a handful of comments from people who will avow that e.g. the npm ecosystem is hot garbage. Seems to me that it is broke and needs fixed.

Works fine for me. Could be better, but 'broke' is a strong word for something I use heavily, daily, with very few problems. Of course HN users love to complain that the npm ecosystem is full of junk, but that's like complaining that most websites are junk. Just learn to find the good ones. Learn to identify high quality packages. Run it by someone more experienced if you're not sure. And if you really can't find a decent quality package that fulfils your requirements, then that's probably a sign your requirements are specific, so roll up your sleeves and write some code.

> Most JavaScript-related threads I see on HN feature at least a handful of comments from people who will avow that e.g. the npm ecosystem is hot garbage.

It's not the "NPM ecosystem" that's hot garbage, it's the Javascript ecosystem. There's a culture of creating micro-packages that have dozens of of often trivial transitive dependencies. With a lot of popular projects, one "npm install" dumps maybe hundreds of packages and thousands of files into a "node_modules" folder that you can't reasonably move out of your source directory. There's also a culture of breaking interfaces and configuration a lot. Many developers growing up in this ecosystem think this is the "normal" way to do things, thus compounding the problem for "future generations".

None of this has anything to do with NPM or the company that runs it. Some of it has to do with NodeJS, which isn't the same as NPM . Another package manager wouldn't solve any of these problems. In practice, NPM works fine 99.9% of the time. That's "good enough".

The main issue with it is that it's too good and successful, therefore it's too easy to publish whatever crap as your library or its dependencies.

That's not really a fault in the tool managing installation and publishing the library, its dependencies or managing the repository, providing the auditing system, script runner frontend and the other things npm does.

You could similarly argue github is hot garbage, because there are so many crappy projects hosted on it.

It is the fault of the tool though if previous versions of a package are mutable, indeed deletable.

That's a fault of npm - _the registry_'s policies, not of the package manager. You could do the same with most other popular package managers if you have control over the registry.

Sounds like you're agreeing with the Entropic people: https://twitter.com/bitandbang/status/1134872073896169472

Their approach is to create a different kind of registries, and inevitably they need a new client that supports them.

Yes, changing the registry model is key.

I'm not quite convinced by their federated approach though. It feels like its just spreading out the problems, and not really preventing them from happening, and in the worst case even creating new ones.

As pointed out in the talk, running a registry becomes expensive once it becomes popular. So now instead of 1 central registry (which I agree is not a good idea) needing to fund the hosting, you have maybe 10-100 federated registries, with each one of them needing to fund the hosting and coming up with different economic models around it.

I'm also not sure how they would really be able to ensure immutability of packages in a federated system. A node could simply publish two different packages with the same name and version number to different parts of the network. Yes, you can reduce the impact by saving an integrity hash in a lockfile, but npm already does that today.

> It is the fault of the tool though if previous versions of a package are mutable, indeed deletable.

The package-lock.json contains hashes to verify integrity.

For legal reasons, you can't really have packages that are immutable or undeletable. In practice, that's just not really a big deal.

Also, how do you solve the problem where a maintainer hands over control of the repository to some malicious actor? What if a maintainer gets hacked? Ultimately, you'd have to audit every single release of every package.

Again, despite those theoretical risks, despite the fact that Javascript projects often have thousands of dependencies, it all works out okay in practice.

It can and should be fixed in a backwards compatible way.

>Don't fix it if it aint broke should be a motto for more developers.

npm is broke, though. Better to have a replacement ready for when they finally go bang under the pressure of venture capitalism.

That's what I was about to say, that the current state of JavaScript package management is "broken" in certain aspects. The dependency on a private organization for open-source packages makes no sense.

I think it's a healthy sign that we're starting to see more npm alternatives, decentralized package management protocol/standards. After some churn, may the best ideas win!

The thing I don't get, is, I'm using Git repos as source for my packages for years now, why can't everyone do this?

Not everyone tags releases so you're stuck if you need a fixed version of a dependency.

Or you can tie it to Github and then download a release from their CDN.

You can indeed reference a git repository with a single commit specified, like described in the npm install docs[1]. Not very user friendly, but it works.

Example:

"dependencies": { "myprivatemodule": "git+ssh://git@github.com:user/project.git#commit-hash" }

[1] https://docs.npm.red/cli/install.html

This! plus treating every commit as major version and be done with it.

That's a bit too idealistic; in practice, any commit MAY have a bug which passed quality control. The problem is that it's all down to people's discipline, it's not an enforced standard.

A package manager where every release has to be reviewed, tested and approved before they become generally available would be a pretty interesting case, I know bigger companies who are reluctant to upgrade because of known bugs in the past would be willing to pay for something like that.

I'm pretty sure you can use a commit hash as a version, too.

Because versioning & dependencies are a nightmare that way

I have only done this partially over the years, but yes I agree, it's a viable alternative and when it comes to private internal modules you don't ever have to worry about singing up to npm enterprise or deploy Nexus or equivalent.

Because package managers are used for more than just source code - many packages want or need to distribute intermediate or compiled artifacts.

There's quite a bit of hand wringing about whether or not this is good or bad as if there's no universe in which Entropic and NPM simply co-exist. Options/competition is good for tools like this, why not encourage it?

On the "new registries" note, though, and I might be alone on this one, I'm honestly the most interested in Github's package registry. No, it's not decentralized, yes it's still owned by a big company, but I'm personally fine with the tradeoffs there. It's kind of reassuring that the entirety of Github's revenue is a rounding error for Microsoft, so I at least don't think there's the same concern around the VC backing of NPM.

As "life finds a way", I believe packages will find a way too.

If npm stops being the go-to solution, another one will take its place, naturaly adopted by the community, following the path of least resistance. Centralized or not. It does not really matter. What does matter is the code that's being downloaded. Modules dependencies management is an old problem, countless of tools have tackled.

Is the node and front projects architectures that dependent on npm ? It's a little coupled but not that coupled ? The dependency management still require a little brain power from dev teams, or does it really fully relie on the package managers ?

Forgive me for being a contrarian but I see nothing more than yet another Stallmanist rant in this?

Ceej and company are trying to portray NPM as the villain here, but is the entire JS ecosystem's burden their's to carry? They keep whining that Roald Dahl is miserable and all despite inventing Node and why the NPM founder is so rich with VC money, etc. According to me, therein lies their hypocracy:

On one hand, you declare your altruism and how you care about commons and not money but on the other hand, you try to imply guilt on those who try to earn money out of JS ecosystem! How is this fair? You do your job and let them do theirs, I don't see how NPM is made the villain here?

My point was : node is just a web server and npm just a package manager. There was countless before, there will be countless after, we will continue to code with or without. I have worked with a few technologies and in a few industries and package management is always complicated. The nature of the package management tools are not that important.

The passion that this kind of topic seems to unleash looks like the consequence of a terrible fear a loosing control over an irreplaceable tool. But no tool is irreplaceable : if you can't/don't want to(for political reasons) use node, npm or even javascript just use something else.

That just it. I have no opinion about who makes money from what. At least, not in this thread.

Minor note: it's not accurate to say that node is a web server. Web servers are frequently built with node but node is also used for CLIs, graphical applications, parsers, compilers, etc.

I see. Thanks for the clarification

> They keep whining that Roald Dahl is miserable and all despite inventing Node

I think you mean Ryan Dahl haha

I think it's less that npm is making money, but that they've taken VC money.

Venture capitalists don't care about the community, or the fact that they're funding critical infrastructure.

They care about extracting rents in order to get back 100x what they put in.

Do you not see the blindingly obvious problem here?

Feels like there's a piece of the story missing:

"There was one individual who planned though and didn’t give their viral software away for free."

Who was that?

Isaac Z. Schlueter, the original creator of npm? I thought it was clear from the context but perhaps I'm missing something.

What about Yarn?

Yarn currently uses the npm registry.

As a Facebook-managed project, Yarn's focus is probably more on FB's needs: fast, repeatable builds from centralized package repositories on their own private network.

There's ample room for a federated community project in this space, IMHO. I'd certainly be eager to kiss npm goodbye.

Entropic is about the registry, not the CLI.

One of the slides says:

> they hired a CEO who made some, um, interesting moves

What is this referring to?

Previously, https://www.theregister.co.uk/2019/04/01/npm_layoff_staff/

Yesterday, https://www.theregister.co.uk/2019/06/14/npm_union_busting_c...

Instead of using a whole new package manager, couldn't people just use IPFS through npm?

I am wondering, without having much knowledge about it, could a system like Ubuntu's package manager & registry (or any other Linux's distro) be suitable for JS?

Of course. APT is used to install JS libraries already but people prefer the npm because it gives people enough freedom [to make the ecosystem a complete mess].

Can you please explain how APT limits freedom?

I think it only limits "freedom" in that it only makes sense on Debian-based operating systems. Whereas I can and do run npm on tons of different Linux distributions, MacOS, and (theoretically) Windows.

You can do things with deb packages on other Linux distributions kinda-sorta using various programs and scripts, but it's not painless. And Mac and Windows are right out.

The same applies generally as to why there are usually two types of dependency managers. One for binary packages and binary shared libraries (even though these often include source code for C & C++ stuff), and one for specific language development environments. See also Rust, Dart, Python, Ruby, Go, Lua, etc., etc. All those languages have their own package managers.

I take it to mean that with npm there is a very low barrier for entry. I mean it's not difficult to actually host an apt repo on your own server or on S3, but there's still more you need to know to get to that stage. I personally think that using something like apt for Node.js dependencies is a very good idea. I work with Node at work and on the side, and my/our usual stack is either Debian or Ubuntu. Even when Docker gets involved it's usually a Debian or Debian based image. [Edit: phrasing]

I am trying to understand why we make NPM such a big deal (and suddenly have all these projects born to solve the NPM problem) when people have solve the issue of a package repository already. C j Silverio's point (JSConf's talk [1]) was that is NPM's scale that makes it hard.

Refs:

1: https://www.youtube.com/watch?v=MO8hZlgK5zc

To put some perspective on this, comparing Debian to npm:

Number of packages:

Debian has 172,000 packages for the most popular architecture, amd64. i386 has 24,000 packages. The rest have less than 500 each.

npm recently broke a million packages.

Download counts:

I couldn't find numbers for Debian. npm served 11.2 billion downloads last week.

(Debian numbers retrieved by https://popcon.debian.org/, npm numbers from https://medium.com/npm-inc/npm-weekly-200-dont-miss-today-s-... and https://api.npmjs.org/downloads/point/last-week

Debian also has a repository model which allows the packages to be served by a static file server, because the dependency resolution is entirely client-side. Npm doesn't.

Hm, as far as I know, resolution is also entirely client-side for npm. Do you have any links about this?

Partially self-inflicted since a normal JS project will redownload the same packages over and over again, hundreds of times throughout their existence.

By default it does not allow to distribute and install multiple versions of the same library, and for good reasons.

Reply on news.ycombinator.com