IPFS is a new peer-to-peer hypermedia protocol | Hacker News Reader

IPFS is a new peer-to-peer hypermedia protocol

ipfs.io

244 pointstambourine_man74 comments

(EDIT: we've a meetup coming up soon. The SV Ethereum Meetup is hosting an intro to IPFS later this month in San Francisco: http://www.meetup.com/EthereumSiliconValley/events/221350594... come hang out!)

--------------- Hello Everyone! ---------------

@jbenet here. Thanks for all the attention! Did not expect to wake up to this post today :)

The best resources for learning about ipfs are:

- The IPFS Paper: http://gateway.ipfs.io/ipfs/QmR7GSQM93Cx5eAg6a6yRzNde1FQv7uL... (note: DRAFT 3, working on DRAFT 4, lots has evolved.)

- The Alpha Demo: https://www.youtube.com/watch?v=8CMxDNuuAiQ

- The First Tech Talk: https://www.youtube.com/watch?v=Fa4pckodM9g

- The IRC Logs: http://botbot.me/freenode/ipfs/ -- there's so much knowledge in these, we're working on moving it over.

We're working on a bunch of more expository media:

- Specs!! https://github.com/ipfs/specs

- More Demos

- More Talks

- Draft 4 of the paper

But there's only so many hours in a day, and we spend most of our time developing. All the resources will be way more polished as time goes on. Please bear with us as we make these available. In the meantime, ask questions in our new FAQ (below), or in #ipfs on irc.freenode.net -- And we'll be answering questions here too of course.

I've just created: https://github.com/ipfs/faq -- if you ask questions there we can build a knowledge base

IPFS already works-- you can install it and try it out. It's not production ready yet, but it's already very reliable. I use it to move around all sorts of personal files, and we use it to host many websites.

Some people asked about what applications are there now? There's many static websites on ipfs, and some simple webapps. We ship our entire ipfs webui with ipfs itself (turtles all the way down!). In particular, Eris Industries recently made a decentralized YouTube clone using IPFS and their Decerver: https://github.com/eris-ltd/2gather If you see more-- please let us know at https://github.com/ipfs/community so we add it to an index (we just reorged our repos under the github org last week)..

We need to make some more tooling to make web-publishing with ipfs really, really nice. We'll be making that over the next few weeks, so if you're interested in pushing out websites with ipfs-- come talk to us.

Another big use case we're going after is Docker container distribution. We're preparing a whole bunch of demos right now that we'll show off in the next week or two. You can peek at: https://github.com/ipfs/container-demos We can already boot entire VMs and containers over ipfs in <10 seconds :D which makes me really happy. You can boot to ipfs with this makefile: https://gist.github.com/jbenet/76afa70955910a2ed097

If you want to help, the best ways are:

- Come hack with us! We're particularly looking for Go hackers

- Just use it!

- Write applications on top.

- Report issues: https://github.com/jbenet/go-ipfs/issues and so on.

- Help us with docs

- Write another implementation (this is blocked by https://github.com/ipfs/specs/)

Some ipfs community members i've seen posting below are: whyrusleeping, dylankpowers, inconshreveable

Also, IPFS development is sponsored by our company, "Protocol Labs, Inc" -- we also make Filecoin. We are hiring, so check out: http://ipn.io/join (yeah-- we just changed the company name. still need to update ipn.io / get a new domain.)

See you around the tubes! :) - @jbenet

Congratulations on a nice talk! And thanks for all this information.

Their protocol github repository [0] is much better at explaining the project than the website.

"IPFS is a distributed file system that seeks to connect all computing devices with the same system of files. In some ways, this is similar to the original aims of the Web, but IPFS is actually more similar to a single bittorrent swarm exchanging git objects."

[0] https://github.com/ipfs/ipfs

Edit: a more compelling summary from an unfinished paper: "The InterPlanetary File System (IPFS) is a peer-to-peer distributed file system capable of sharing the same files with millions of nodes. It features a completely decentralized architecture, secure and efficient peer-to-peer block distribution, and a path-based naming system supporting distinguishing mutable and immutable names. The Web today still uses HTTP as the main data transport. IPFS is capable of evolving the web to take advantage of versioning, p2p distribution, cryptographic operations, and decentralized publishing. Moreover, it presents an opportunity to construct a web whose links do not rot, whose files are deduplicated globally, and whose websites are no longer “sites”. IPFS is a step toward The Permanent Web."

A PDF of paper quoted, 'IPFS - Toward The Permanent Web (DRAFT 1)', may be found here: https://github.com/ipfs/papers/blob/master/ipfs-ttpw/ipfs-to...

If you're looking for the in-depth technical detail of IPFS, you really want to read Juan's paper on it:

https://github.com/ipfs/ipfs/raw/master/papers/ipfs-cap2pfs/...

I've explained IPFS to friends in the past:

Imagines IPFS as running Git where all of the objects (files, directories, commits, etc) are available over a BitTorrent swarm. All files are accessed via their checksum so they have cryptographic integrity even over insecure networks. Then imagine you can mount it as a file system or browse it via HTTP, and run your own node to seed files.

Popular files will be seeded by more people, making them quicker and more reliable to access. As long as clients are still requesting a file, it will continue to be seeded even when the original publisher goes offline.

Pitching a developer product is hard. Traditional marketing advice encourages you to pitch value because the implementation details are usually not important to your customer. Developer products require a more delicate balance because ultimately we have to work with the technology and want to understand what its real limitations are.

These ideas, and how they combine, is natural and elegant, even inevitable.

For example, the prototypical idea of swapping out Git's disk-trie content-addressed object store for an actual DHT to get immutable distributed file system must have occurred to a lot of people when they first learned about Git's internals.

I don't know about @jbenet's particular journey to these ideas, but I suspect Rich Hickey deserves a lot of credit for using Clojure (and his fun, smart talks) to popularize immutability and name/value separation for a new generation of programmers. Even just ambiently.

That's certainly how I started realizing the power of that paradigm, even though I was already using it for my day job! Especially when I then saw it used building Datomic, which of course leverages immutability to totally refactor the traditional database architecture from client/server into self-serve-client/server/transactor in a really interesting way.

Also, with Clojure's HAMTs, you've got hash trees and cheap immutable update, and that again primes you to read about (or reinvent) Merkle trees. So I doubt nearly as many people would know about such neat CS things without Clojure.

Actually, Bitcoin is also a really obvious way these ideas would become more widespread, though I don't know how many people really go through the trouble of understanding the blockchain mechanism.

And Linus, of course, for Git. Git internal design is really nice and everyone should dip in at some point in their career!

We should give Ralph Merkle a Turing award -- that single idea unlocks so much. Except we shouldn't because he patented Merkle Trees, apparently.

> I don't know about @jbenet's particular journey to these ideas

Git is the single largest influence on me. After that, various other papers (all the CAN literature, bitcoin/bittorrent, CRDTs, etc), and conversations with people. in particular, two close friends (jzim, davidad).

> We should give Ralph Merkle a Turing award

I agree! Zooko recently told me that a long while ago they dubbed themselves "Merkle Tree Huggers". I am too!

> Except we shouldn't because he patented Merkle Trees, apparently.

Sigh. patents... :(

> Git is the single largest influence on me. After that, various other papers (all the CAN literature, bitcoin/bittorrent, CRDTs, etc), and conversations with people. in particular, two close friends (jzim, davidad).

Oh, interesting.

Funny coincidence: I actually met davidad once, when I first starting out at my job. I think because of his asynchronous logic automata. We somehow ended up discussing busy beaver Turing machines over ice cream†.

† Toscanini's near MIT, fantastic ice cream.

davidad is one of my favorite people in the world :)

> Toscanini's near MIT, fantastic ice cream.

Much agreed.

Please never ever ever implement a Merkle tree as you did in https://github.com/jbenet/random-ideas/issues/20 . This is vulnerable to the same trivial collision attack as the original Gnutella THEX proposal.

EDIT: also, please implement non-power-of-2 leaf node Merkle trees as power-of-2 lead node trees with the rightmost leaves containing the hashes of the zero length array, that way the final block along with its hash chain to the root provides a cryptographic proof of file length. This prevents denial-of-service mischief where a malicious server claims a given resource is terabytes large and a poorly written client starts allocating TB of disk space on a filesystem that doesn't support sparse files.

Or, more efficiently, at each level of the tree where there are an odd number of nodes, add another leaf node containing the empty byte array on the right side of the tree to make an even number of nodes.

I just came across Filecoin on their Github readme FAQ and it sounds like a great idea to me:

"What is the relation to Filecoin?

Filecoin is the sister protocol of IPFS. While neither protocol depends on each other, they are perfectly complementary. IPFS handles high performance distribution of objects; Filecoin handles incentivizing the world to backup/seed objects."

Cryptocurrency backed by real, useful, and valuable commodities (storage and bandwidth).

Didn't Mojonation, from the Evil Geniuses for a Better Tomorrow, originally have plans kind of like that?

kindof, yeah. Zooko wanted to build a distributed data economy with mojo, i'm not too sure on what happened to it, but we have talked with zooko and other people on the tahoe-lafs project and they seem interested in working out some sort of interop between our systems.

I believe mojonation's currency was not decentralized

It wasn't, but you could have had multiple issuers of currency. Not that it ever got big enough to flex that part of the protocol.

This blog post, which summarizes a talk on IPFS, provides a better introduction than the landing page or the Github readme:

https://sourcegraph.com/blog/ipfs-the-permanent-web-by-juan-...

It's annoying that there is no exhaustive comparison to existing systems, such as Bittorrent Sync, but I think that in this case it's genuinely difficult to find something that really resembles IPFS. The closest thing is probably Git, but IPFS maps the entire "global network" to your local file system.

It could be useful for any form of infromation sharing, but the author provides a bunch of "trivial little JS apps" as examples. However, I'm not sure if web apps are the best application for this, since the database would need to be hosted in a traditional way anyways (unless you can exclude it from being version-controlled?)

Like everyone else, though, I wish there was less hand-waving and more down-to-earth explanations on the subject (especially on the landing page!)

The database is Ethereum (for data that needs consensus), your filesystem (for data that only matters to you), and Whisper (for ephemeral data that doesn't need consensus, like messages). You build a JavaScript app that calls into libraries that implement these technologies, and you host your code on IPFS.

The world is currently figuring out the best ways to utilize this sort of architecture. Join us.

https://dappsforbeginners.wordpress.com/

This looks very cool, but I'm wondering about abuse. I watched the video and did some followup research using Google and couldn't find anything covering how to handle abuse within IPFS. While I'm sure there will be a lot of photos of cats and wildlife videos it seems that this solution would be ripe for abuse by the seedier regions of the web. If I'm participating in IPFS how do filter for or stop illegal content from being saved and hosted by my node?

It might be just my opinion, but I think this is a problem with the legal system rather than with the protocol; NTFS doesn't have mechanisms for dealing with "abuse", why should IPFS?

You typically don't open NTFS to the world & if you do, don't be surprised if law enforcement comes knocking. The fact is, there's a lot of child porn on the web as an example and the fact is, if child porn is found on a server you host there's a good chance you will be taken through "legal hell" in many countries as a result.

No one exposes NTFS volumes to the open Internet. A sibling comment got it right, IMHO, that spam, and not illegal content is the real pitfall here.

Seconded.

I'm less concerned about illegal content per-se as I am concerned about spam. I created Neocities (https://neocities.org) and we get quite a bit of SEO and pharmaceutical spam that we need to remove. There would be no way to prevent that with IPFS.

The one thing I could see this being useful for is storage with a private network. If you control all the nodes, then you could filter what goes into them. But I recognize that the point here is to make everybody run on the same network.

Bitcoin has a similar problem. If transactions were free, it would be very easy for someone to just flood the network with garbage transactions and essentially make the blockchain useless for legitimate use. The way this is prevented is by charging a transaction fee (around 7 cents) for transactions that don't meet certain criteria. It's not an ideal solution (because it's fixed in the code and doesn't fluctuate to compensate for market demand), but regardless, it's the only way they've found to prevent this sort of spam.

Without a solution like this, IPFS would not be able to be used in a decentralized, trustless manner. The solution here is probably some sort of monetary restriction, similar to the fees on Bitcoin. But I have no idea how you could implement that here.

That said, if a solution is discovered, Neocities will be front row center to implementing it for our sites. I think this is a genius idea, I've been following it for a while, and I love it.

It sounds like pull instead of push though, so the user presumably is only seeding things they have looked at, so a flag/delete/block button for them to remove and block something from their node might be good enough.

There's a block exchange protocol where that can be built in: refuse exchanges relating to block X. There is also potential for a web-of-trust built on top of that as well. So rather than a centralized flagging system, if sufficient numbers of people you marked as trusted flag something, it's likely not something you want to make block exchanges with.

I'd say this one falls into the "sort of" category. In web security we are constantly fighting cross site request forgery (CSRF) bugs. Basically a bad actor places a GET or POST request into a page you visit which makes a request to another asset you are likely to have visited while authenticated like Facebook. This GET or POST then leads to some action on your behalf without your knowledge leading to an action of consequence such as making a wall post on your behalf. Similarly, within the context of this solution a bad actor could embed links to objects they want you to cache without your knowledge by not rendering the objects in a visible manner in the DOM. Yes, this can happen today and yes, the technique could be used to frame someone. In fact there was presumably once a court case where the defense team claimed their defendant didn't browse child porn and to provide the judge an example that it's possible to have child porn on a machine without ever knowingly visiting such content they had the judge browse the defense team website where they loaded up a bunch of images which went into his browser cache but which he didn't see because they weren't visible in the DOM. They then asked the forensics investigator to see if the judge had browsed the particular content and of course he came back that the judge did.

In the case of this solution, a bad actor could use unsuspecting visitors to a site or users of a rogue WiFi network to load up the particular bad content they want distributed.

All content is self hosted until another party is interested enough to pull it locally. While spam will still be something that needs to be dealt with, spammers will be using their own disk space and bandwidth, and not that of other users.

Your node only retrieves content you request, which can be manually purged.

The only content saved and hosted from your node is content that you have requested yourself. In that regard it is no different from BitTorrent. Any content you request is purged from your node over time unless you have explicitly pinned it to keep around forever. So as long as you stay away from the seedier regions of the web, you will be safe.

Also solves the problem of not wanting to donate your CPU, bandwidth, or storage to things like kidporn. That's a serious issue with things like Freenet, where you have no choice about what you cache/host.

> Any content you request is purged from your node over time unless

Won't this mean that rarely used files aren't really permanent ,which is a known problem with bittorent ?

Yes that's true. Files are only guaranteed to be available indefinitely if someone has decided the file to be noteworthy enough to pin and their system is online. Ideally, systems like http://filecoin.io/ will be built upon ipfs to enable those that care enough about their files to obtain distribution through paying a little money to incentivize others to host their files.

as dylankpowers says here, you only host content on your node that you choose to by default. We will have a feature later on that will allow the network to adapt to what objects are the most popular and rehost them automatically, but this process will respect a blacklist that we will curate, and is of course, optional per user.

Hint to the developers: summary and overview in plaintext, please. I have no idea what this is beyond the title, but it looks interesting.

Here's a thread about IPFS from last year: https://news.ycombinator.com/item?id=8069836

And another from this: https://news.ycombinator.com/item?id=9083849.

Hmmm...

http://blog.bittorrent.com/2014/12/10/project-maelstrom-the-...

We talk with the tahoe-lafs guys a few times, our projects have different goals, and even quite a bit of opportunity for interoperability.

i recently had the pleasure of hanging out with both zooko and warner and they're both excellent. as @whyrusleeping mentioned, there's lots of room for interop + collaboration on things. (in short, we <3 Tahoe-LAFS)

That one is likely relevant too: https://datatracker.ietf.org/doc/draft-ietf-ppsp-peer-protoc...

Can someone explain why all of the hashes in this demo appear to start with the same few symbols ("Qm" in this case)? Ordinarily I'd expect this to suggest that it isn't actually a hash, but perhaps there's some detail I missed.

Assuming it's anything like Bitcoin, it's a network ID prefix separate from the hash.

that's a base58 encoded multihash: https://github.com/jbenet/multihash/ and https://github.com/jbenet/go-multihash/

+∞ for double NAT traversal.

yeah double NATs are annoying :)

So what if I want to dynamically create content? Limited to using this as a sort of cdn edge?

Well, if you need a central source to store and provide the state or content then it's not distributed anyway. The notion of developing on a distributed system is a whole different paradigm than traditional web development.

Otherwise, JS is JS no matter how its received.

I was wondering this as well. Ultimately, distributed systems are just a very different animal. The first step is designing your data to be stored in files, rather than a database. In practice of course, all durable databases persist to disk, so that's not a big deal.

From there though, you have 2 more problems: keeping all data from being public (maybe becoming "friends" with another user involves a key exchange?) and distributing updates in a reasonable way. It'd be kinda strange if Lady Gaga had to buy a ginormous server to serve all the requests to her Twitter feed. I'm not sure how IPFS is implemented but if it's similar to BitTorrent in the sense that data can be seeded from any node, then that problem sort-of takes care of itself. You definitely can't think about "dynamically creating content" in the sense that twitter.com/:username may render different things at different times. The concept of web search also becomes an (even more) interesting one.

On a somewhat-related note, one also has to wonder how this system would interact with the European "right to be forgotten". Any legislature ensuring such a "right" would make this protocol a bit of a non-starter (at least in those jurisdictions).

Ultimately I think IPFS is a pretty cool concept, one I've kicked around a bit but never fleshed out. Major kudos to Juan and everyone else for taking it on - I look forward to seeing where this goes.

IPFS has ipfs for immutable data, and ipns for mutable ones. It uses a DHT to name an ipfs hash.

We're actually in the middle of finishing up a demo for mounting a Docker registry on top of IPFS, and showing we can freely push/pull from it. https://github.com/ipfs/container-demos/pull/19

> The first step is designing your data to be stored in files, rather than a database

Somewhere in the middle-- ipfs gives you native access to the merkledag datastructure, so you can actually port most databases to using ipfs as a kv-store directly. (think an embedded sqlite or something). and on the other side, maybe you dont need a database. maybe you can just construct the graph of objects you need and write that to ipfs.

> maybe becoming "friends" with another user involves a key exchange?

yep! we've discussed an `ipfs friend` cmd

> Ultimately I think IPFS is a pretty cool concept, one I've kicked around a bit but never fleshed out. Major kudos to Juan and everyone else for taking it on - I look forward to seeing where this goes.

Thank you!! Come help us build it! :)

we have a system for this called IPNS, (the interplanetary naming system). It is essentially a global, signed pointer to whatever content you want, we also have DNS mappings to ipfs objects as well.

see section 3.7 of the paper here: https://github.com/ipfs/ipfs/blob/master/papers/ipfs-cap2pfs...

I watched the Alpha Demo and briefly read through the comments on this thread and description on the github, but still am a bit confused.

I mean, there already are multiple file-sharing protocols. There are video-streaming protocols. There are p2p data-exchange protocols. Some of them need more friendly end-user tools to become widespread, but still, they do their job and usually are introduced as having some feature, which "all the previous stuff" lacked.

So what exactly is IPFS and why I would use it over… whatever else I can use to share files? After all, it's a protocol, and not an end-user tool yet, right? So what does this protocol present us? What exactly all that "p2p, decentralized" stuff means in the case of this very protocol?

Why I'm asking it here and not seeking answer in documentation and source-code? Well, there's quite a real reason beyond my laziness: from what I've understood so far, it seems like some pretty ambitious project. And ambitious projects need one short, clearly expressed idea, to avoid becoming a mess during the development process. I don't see one.

is there a way to circumstantially throttle ipfs specific traffic?

also, note the docs' command lists anchor to the wrong page: http://ipfs.io/#ipfs-config ; stead: http://ipfs.io/docs/commands/#ipfs-config ;

we are going to be able to set bandwidth caps soon, most of the code required is in place, just need some more dev time and testing

should have mentioned in the parent.. looks really interesting and feature rich, great job

Can anyone explain to me how the p2p network determines which files to persist, given a limited amount of storage available?

This seems to make many of my infrastructure problems go away for a number of applications I can imagine. One thing I am wondering about though: is there any type of push or pub/sub so I can be notified of updates to some data or set of data?

Currently no, but I'm brainstorming a pub/sub system that will operate through the DHT, its still on the backburner for now as other things are stealing my attention.

Just in case you're looking for references: I think of https://github.com/mnaamani/otrtalk as a pub/sub system over DHT.

@jbenet Interesting stuff! We worked on something related to your work for our master's thesis - including location-independent naming, a WebRTC-based Chord DHT and an ALM/PubSub layer using Scribe. Check out https://github.com/boplish/core and http://inet.cpt.haw-hamburg.de/members/christian-vogt for the publications.

We used to have proxies that cached http requests. And there are already technology for broadcasting.

I do not see how this will be better then say Torrents. Or if anyone would use this for more then sharing "copyrighted" content.

I'm not saying this is bad, just that you can do better at marketing/selling it. You mentioned on the second video that this is targeted to those that do not know how to set up a LAN connection. Then maybe you shouldn't use the CLI in the demo.

While other comments about security may be right, it probably isn't a reason not to pursue this. It's a great idea and very worthwhile pursuing.

My main concern is over using hashes as an address. A hash isn't a unique address but I haven't seen any mention of protections being built in to protect against the risks of downloading different files with the same hashes.

Searching for "ipfs collisions" returns this IRC discussion, in which joeyh makes the same point: https://botbot.me/freenode/ipfs/2015-03-05/#33399953 (search for "collision" if it doesn't appear immediately).

I have a question about IPFS. I don't mean this in ANY kind of negative way at all, but what problems does IPFS solve?

I got really excited about IPFS when I first learned about it, but after to talking to people for a while, I actually made a document titled "What problems does IPFS solve?" and thought for a long time. It's still blank.

I would love to hear from anyone involved with the project what applications this could have.

It solves the hacky kludge that is the state of the web. I suspect this opinion is unpopular, but the web was never designed for web applications. It was designed for static text and images. JavaScript as it is currently used (to drive complex web applications) is an abomination. Browsers and their monstrous, labyrinthine, all-encompassing codebases and extension ecosystems and security problems are an abomination. They're also broken in every way in terms of security. Nobody's responsible; the current state of the web is just the accumulation of two decades of layering technology upon technology and making the mistake of continuing to layer where we should have built something new. The initial discovery that web apps ran everywhere by default kind of prompted a tidal wave of development just for the web that few ever bothered to take a second glance at. As a result the browser became an operating system.

The web was not designed for web applications.

IPFS can fix this. Instead of putting all our shit in the browser, with the steaming mess of CSS, JS, JS frameworks et al, now you can write native clients that use IPFS to provide the same kind of functionality the Web does. The infrastructure is there.

The problem is that nobody wants to write a native client for every device. That's why the browser was popular to begin with -- it's an application as a runtime, an application that everyone knows about and anyone can install. That everyone does install. IPFS doesn't solve the problem of having to write native clients more than once for different platforms. But it sure as hell solves the hard part, which is the networking part. At least I think so.

So, what's an example? It's really hard to have dynamic content with IPFS because it doesn't really have the concept of a "request", only a lookup-by-key type of thing. For example, consider how difficult it would be to make a simple messaging app with IPFS.

Frequent-ish polling of a file will keep looking for it in the nearby nodes. I suppose it would be smart enough to build up a network around a specific file. If it's close enough to real time, it could be a real solution

It's true that works, but which part of that is an advantage over the web the way it is now?

Finally! Our file sharing efforts are no longer bound to Earth

I don't see a way to install it on Windows.

> Gateway Timeout

They should've hosted on IPFS!

There's a mirror at http://gateway.ipfs.io/ipfs/QmTjgwrstCTTWCe1BvoMm5hmX7GMj1q1...

I think they already do: "There is a mirror of this website hosted at http://gateway.ipfs.io/ipfs/QmTjgwrstCTTWCe1BvoMm5hmX7GMj1q1... We plan to transition to hosting it on IPFS as soon as DNS -> IPNS -> IPFS naming proves robust, so we can use nice looking URLs."

Reply on news.ycombinator.com