Show HN: EverythingStays – Immutable and Distributed Node.js Modules with IPFS | Hacker News Reader

Show HN: EverythingStays – Immutable and Distributed Node.js Modules with IPFS

everythingstays.com

103 pointsdiggan49 comments

Why not GX? https://github.com/whyrusleeping/gx

(It's a language-agnostic packaging tool built around IPFS)

To not reinvent the whole package mananger. We're trying to reuse the existing NPM CLI for things and only change where the packages comes from.

Having something that works with both existing package.json dependencies and an extra field in the package.json, is the goal.

People should be able to use IPFS when it's available, and fall back to NPM central registry when needed.

Does gx support multiple, parallel versions of the same dependency (e.g. a dependency tree)? How does it work, and does it work with Node.js's require/resolve algorithm?

author of gx here, gx packages are referenced unambiguously by the hash of their contents. You can have multiple versions of the same package without even having to think about it :)

Why can't GX and this tool store packages in the same format, so one can use packages from the other?

Nice initiative!

Just want to tip you and others about IPM - Immutable Package Manager.

We are a group of people trying to do just this, we have a lot of healthy discussions going on if you'd like to collaborate or just merge these initiatives.

We have a couple of people investigating using gx which is an universal package manager based on IPFS, we call it gx-node.

We also investigate if there is a quicker way, wrapping npm instead to disable unpublishing of npm packages.

Please check it out over at http://ipmjs.org if you are interested.

Thanks! I actually was ipmjs and I was surprised on how wrongly focused the attempt is, and my reaction was to write down my ideas with EverythingStays. The "no politics" part is there, only because I was reading the ipmjs/ipmjs issues...

I think the idea or rewriting what already works, is an wasted effort. Rather, having npm to just execute shellscripts to get the files from IPFS, JustWorks(TM)

Also, there is discussions about malicious packages and copyright infringement, something that is not a problem since it's up to people what they want to seed. And setting out to "solve" npm scripts...

Basically, if you want to reinvent everything what NPM already built + a little bit more, ipmjs seems fun.

If you want something practical and pragmatic that works today, EverythingStays is for you.

  "I was surprised on how wrongly focused the attempt is"

I'd hope you would have added your 2 cents to our discussions since it would have been be really valuable. And you still should because there seems to be two main road maps for IPM right now, if I understand the discussions correctly:

- extending npm, adding a immutable dependencies (similar to EverythingStays)

- extending gx to add npm-like functionality (re-invent npm in IPFS-land)

I am gonna add an "issue" in our repo referring to your project, and hopefully, we could somehow merge so we don't implement the same thing twice. :D

Make sense?

Yeah, I was about to, but then I saw the owners of the project agree to some of the things I absolutely want to not touch at all. So my thinking is that our ways was different enough, that my feedback would not be valuable to you.

Feel free to open the issue and I'll chime in with what I think. Then we'll see if we agree or not :)

For me, it never was about what I thought, but what the community thought and wanted. My goal was just to create an discussion. I am sad that the introduction didn't make that more obvious so you would have felt more invited to join the discussion. My bad.

I think several people agree with you in our discussions, and the more I think about it myself, the more it make sense, short-term to extend npm and add immutable dependencies.

I, personally, like the idea of gx, but that is a more of a long-term achievement, developing an open-sourced, fully decentralized , platform independent package manager. :)

Here is the newly created issue if you want to shime in -> https://github.com/ipmjs/ipmjs/issues/13

> Yeah, I was about to, but then I saw the owners of the project agree to some of the things I absolutely want to not touch at all.

Can you elaborate on which specific areas you're trying not to touch? In your previous comment, you mentioned "rewriting what already works" and "malicious packages and copyright infringement"; anything else?

Things that was raised in the ipmjs repository that I don't feel the a package manager should solve

> Prevent worm-like vulnerabilities when adding dependencies

> npm scripts, the good, the bad, and the dangerous

> Namespace all the things!

> Needs for frontend and backend package managers are different

> Involve the FSF

> Have you thought about malicious packages?

> Have you thought about copyright infringement?

> trademarking 'ipmjs' so we don't get kik'd

These are all issues I want to avoid. Also, I don't want to write yet another package manager, I'm trying to extend NPM and still allow the centralized way that we currently use today, but allow a optional offline-first approach.

Feel free to continue reading more about what I said in the ipmjs repository here: https://github.com/ipmjs/ipmjs/issues/13

If you want to build a solution that lasts, building it on top of a new, unproven and not widely adopted technology like IPFS seems a risky choice at least. It is already not to easy to predict what will happen to a technology in the next five or ten years when it has has been around for decades, in case of something as young as IPFS you can probably just flip a coin.

TeX, Perl and Erlang have CTAN [1], CPAN and CEAN respectively and are now almost 25 years old. I would say the most important thing is having a large number of mirrors and it seems to me that mirroring a set of packages with an ordinary web or FTP server lowers the barrier to entry quite a bit. You will of course have to roll your own synchronization and discovery mechanism on top of that but it has been done and works.

And one thing that most likely won't work anyway is creating a distributed solution, releasing it into the wild and the problem is solved. Someone has to take ownership, issues will come up and someone has to take care of them, even if it is only fixing bugs or adapting the code to changing environments.

[1] https://en.wikipedia.org/wiki/CTAN

Anecdotal: been running IPFS for a year, and it is simply great. Its my go-to for file-sharing these days, it just works so well. I don't think its going to go away any time soon ..

That really doesn't change that it has zero adoption. It's hard to quantify but I just searched the programming languages ranked 90 to 100 in the TIOBE index on Google, they have millions of results, IPFS brings up somewhat between 100 and 300 thousand results. Have you ever heard of Andrew File System [1], a distributed file system developed in the 80s? It is certainly not a widely adopted technology as far as I can tell, but it still has ten times more search results.

[1] https://en.wikipedia.org/wiki/Andrew_File_System

Not sure what you're talking about? IPFS[1] has 291.000 results, and "Andrew File System"[2] has 46.400 results. My guess is you searched for Andrew File System without the quotes. Which will return plenty results utterly unrelated to any kind of file system.

[1] https://www.google.com/search?q="IPFS"

[2] https://www.google.com/search?q="andrew+file+system"

I did indeed not add quotes, I only skimmed through a couple of pages to see if the results are relevant but you can not go beyond 1000 results so I certainly missed the irrelevant results.

But that doesn't really change the picture, for me "interplanetary file system" has 5,910 results, "andrew file system" 46,300. The abbreviations IPFS and AFS have 289,000 and 27,700,000 results but both have several meanings and are therefore not representative. If you look at Google Trends, IPFS began rising as a search term in 2011 [1] well before the inception of Interplanetary File System but I failed to quickly figure out what it refers to.

[1] https://www.google.com/trends/explore#q=ipfs

The curve for "ipfs corporation" shows a similar development since 2011 [0], so that seems like the most likely candidate. I would still attribute most of the growth since 2015 to the InterPlanetaryFileSystem, so maybe a third of the search volume.

[0] https://www.google.com/trends/explore#q=ipfs%2C%20ipfs%20cor...

How things grow: people use them. They don't stop using them. IPFS is growing.

CTAN has a central server. CPAN has a master site. What you're proposing has exactly the same deficiency as the npm repository: it is centralized and thus vulnerable to takedown notices, "right to be forgotten", author deletion and similar shenanigans.

IPFS is distributed and has no such single point of failure (SPOF).

You don't have to allow deletions or replicate them. I also see no reason one could not send out takedown notices to IPFS nodes, especially in the current situation with a pretty small system - I found a number of about 2000 nodes in January 2016.

That's fair critique, but at one point you have to start implementing things with a technology, to help the technology grow and get real world usage.

Yeah, I agree. My point of this adventure is to have a large set of mirros, that are easy to setup.

> You will of course have to roll your own synchronization and discovery mechanism on top of that but it has been done and works

I'm not interested in implementing these things, and would rather compose existing parts to achieve my goals. IPFS is just one part of the puzzle.

You're right about the ownership. Everyone is dependently owning the issue of seeding their own modules. I will provide a public mirror that people can use if they want or not, and maybe some other organization steps forward to help with mirroring as well (npm maybe?)

How are you handling security-related issues such as user identity and IPFS namespaces, particularly as they relate to systems outside of IPFS (i.e. github)? Is there a method by which someone who has lost control over a namespace may announce that unfortunate situ, and then form a new, "valid" one?

Also, regarding the ability to "remove": could you clarify what this means? Given the nature of IPFS, is this really just the signaling the deprecation of a version or entire module using data posted to IPFS?

I'm not exactly sure how you mean. If I publish lodash version 1.0.0 and get the hash "ABCDEF", that will always be the hash and you'll always the same contents. There is no sense of losing control over the published package. You can although lose control over the way you are distributing the hash itself, but that's a different problem.

> regarding the ability to "remove":

I'm not sure what you mean, could you refer the sentence where you see this, so I'm sure to respond in a good way.

With IPFS, there will be no deletion or removal, how bad you may want it. The nodes you can control, you can tell them to stop sharing it, but the distributed nature of IPFS makes it impossible to guarantee that other people won't share it.

Heh, I saw the "able to remove modules" under Problems Trying to Solve in https://github.com/EverythingStays/EverythingStays and thought you meant that was a feature not a bug. I was definitely referring to control as opposed to removing a single version (impossible by design in IPFS).

Conceptually though, the use case of remove is important, at least in the sense that an author wants to formally call out when a version contains breaking code, etc. I am wondering how an author can signal that the latest version is malware or should be ignored, or that the whole namespace, or even the author is no longer valid.

Without human intermediaries (the primary benefit of centralized systems), we lose the ability to mitigate or moderate when things go awry -- e.g. an author loses control over the "keys to the house".

As this concept gets developed, it would be good to develop protocols for identity management, release validation, module namespace registration, etc. Perhaps if not in IPFS, it can be implemented in blockchain tech like ethereum.

I hope I am being more clear here! Let me know :)

There's a significant misconception in this article:

    Today, the fate of a module is decided by NPM the
    company, and they are not afraid of deleting your
    module and letting someone else have it.

This is not what happened in the recent case, it was the author who deleted (unpublished) the module after the name (and the right to publish future versions) was taken away. If he hadn't unpublished them the already published versions would have stayed around indefinitely.

I'm not very interested in what happen with the kik module and whatnot, more interested in the fact that that can happen.

With that said, npm did pull the ownership of kik. Think what you want to think about that, I think that shouldn't be possible.

That the author removed his own dependencies is true and you're right. The text doesn't convey that good enough and I will improve it.

Thanks a lot for the feedback.

Might be interesting for you to know that I've updated the website and removed that part.

Hello everyone! Thanks a lot for the interest, I have learned a lot from the comments you left.

I do like NPM as it is today, except for the central registry. I've made a huge facelift of the website, and also updated the content to reflect this.

It now contains better example, and easier to digest information.

Please take a new look and let me know what you think. http://everythingstays.com

Side question: what is NPMJS Inc's business model?

Seems to be around private modules and a on-site hosted registry, called enterprise.

> a module should live on, for as long as people are seeding the module and using it. More popular modules will be seeded by more people, making it faster to download.

So if you happen to use an unpopular module, or if your project ages, it's quite possible to literally lose your dependency. Package management that is built like this is almost useless. There has to be a central node of some kind.

Or you can just seed it yourself. Or pay someone else do it.

Say you and others believe a full archive is needed to prevent that kind of problems. You set it up any way you would a centralized node, except it isn't centralized, it's jut another node in the network.

Decentralization has some disadvantages, but in terms of availability, I don't see how it can be anything else than strictly superior.

Yes. This is exactly what the current centralized package repo does: seeding of data. With decentralization, anyone can do it instead of one single entity.

I agree of course that a decentralized system is technically superior to a centralized one. It must be, because as you pointed out, strong consensus in the network about a particular node makes it indistinguishable from a 'central' node in a centralized system.

However, when it comes to package management I think we should look at the community first, technology second. The first concern is to set up a central node that everyone can safely mirror and is pretty much guaranteed to be there. Otherwise, what's the point? If I can't trust a package repository to store a package for a decade or so, why even use one?

What's lovely about CPAN and friends is that they are reliable. They're always there. Of course I can just host a node myself for the next 10 years, but that doesn't make everythingstays a good alternative to npm. If all we're doing is decentralizing packages with the assumption that people will host their own stuff, this could be implemented with BitTorrent plus a text file containing magnet links.

Others in this thread mention WebTorrents specifically, and it is true, we could use torrents to solve these problems. What benefits could that provide over IPFS or npm? Does the money that backs npm or a CDN legitimize their ability to last a decade?

You are already hosting your own dependencies, man.

"Community first, technology second"? Do you mean this project only exists to use a cool technology and doesn't care about community? This thing here exists to try to solve a NPM problem. It is community first.

No, the whole point is to not have a central node.

However, it is possible that NPM Inc decides in the future to seed all public packages, but not being able to remove ownership of the packages. Or another private company might come along and do that.

If you use an unpopular module, and you want it to exist, you better seed it, somehow.

The es-cli will help with this in the future, allowing you to spread your module across public and private nodes.

If I use a package manager over decentralized systems (like ipfs), and a decentralized system use hash instead of version number.

how I can automatically update to the last version of a package?

how the package manager know what is the last valid version?

Let's say though that someone publishes a trojan module? And for a while it becomes used in a bunch of other modules or what not, how does one handle that situation? In that case removal or blocking of the malicious module is warranted.

It is sort of like revoking certificates I guess.

The same way you'd handle a trojan module in a statically-linked project; you remove the trojen and republish. It's a issue with linking rather than dependency satisfaction.

Fortunately, because everything is a cryptographic sum, it would be trivially easy to state "IPFS Hash Qm....... is a trojan, please sanitize this hash on all machines".

I would expect things like this coming from CERT or other security researchers in the coming months/years.

What is missing here is an understanding of IPNS, which sits on top of IPFS and allows us to rename nodes and redirect things if the circumstances warrant.

https://groups.google.com/forum/#!topic/ipfs-users/fr6dlQ8we...

Someone publishing a trojan module can be solved in a number of way. Probably the simplest way is to just republish the module and tell people the previous one is bad. You can also provide an optional blacklist (similar to how adblocking blacklists works) where the bad hash would be if needed.

With signing, you would revoke the keys used for the signing.

Would love to see something similar based on WebTorrent

Can you elaborate on the advantages of using WebTorrent instead of IPFS?

You could do wacky things like checking out npm modules by torrenting inside the browser, which I don't think IPFS can do yet. Otherwise I think they're pretty similar.

Npm module for ipfs plx?

Reply on news.ycombinator.com