_prometheus

BTW, we have MaxMind's GeoIP Lite database (we asked first) stored directly on IPFS, indexed with a b-tree. see https://github.com/ipfs/ipfs-geoip :) -- it's what powers our webui globe.

we do not yet measure latency, we really need to and have been wanting to for some time. could you help us with this?

IP address signals are good, but we must maintain real measurements of all our connections to be able to adjust over time. also, with things like cjdns and tor, IP addresses wont help much.

I'd love to get network quality estimation at various levels -- as close to the wire as we can for proper measurements, but also across layers of the stack to quantify overheads and so on.

Would love to see how transports like QUIC improve things like parallel fetching of blocks.

Thanks for the interest everyone! We think libp2p is one of the most exciting and useful parts of the IPFS project. We've refactored it out of IPFS for the benefit of the general p2p / networking community.

We're working on cleaning up interfaces, getting to a good level of completeness and interop between go and js, and then messaging/explanations. Stay tuned for proper a launch later on. But note both go-libp2p and js-libp2p are usable and in use now. :)

- Go: https://github.com/ipfs/go-libp2p

- js: https://github.com/diasdavid/js-libp2p

- specs: https://github.com/ipfs/specs/tree/master/libp2p

brb, 8hr flight.

Thanks! Many answers:

- storing is super cheap. seeding not v expensive. i think the entire npm reg is <1TB?

- (soon can ship entire registries in USB keys!)

- many individuals and orgs should keep/replicate what they depend on

- tools like ipfs-cluster will help organize nodes like a RAID array

- can pay services to back things up (on service infra in vogue)

- can pay _the network_ to back things up -- see http://filecoin.io

jbenet here. We're all very excited about the dev interest in gx and ipfs! thanks! we did not expect attention on it this soon! Please check it out, try it, and give us feedback!

Please know though that things are still super early, and very rough! We do not think our pkg mgment efforts are UX ready for end users who want something strictly better today. But they are ready for early adopters who want to think or hack on them with us. In fact, we are now self-hosting go-ipfs in gx! https://github.com/ipfs/go-ipfs

Something to bear in mind: we are building infrastructure you can rely on long term, with decentralization, flexibility, and futureproofing from the ground up. We want our protocols and tools to survive independent of fragile organizations or fragile routing. We can't take many shortcuts (like depending on centralizing agents), because we want something rock solid to last for decades. This means it takes us a lot longer to build high perf and high quality UX, because there's a lot more to do. More work, but it is all achievable. The rough edges will disappear as we improve the tooling. Even today, gx is amazingly simple and powerful, and already does much for dependability.

It's worth mentioning other package management tooling we're working on too:

* NPM + IPFS - https://github.com/diasdavid/registry-mirror and other repos. This is an effort to improve how NPM works with IPFS

* pacman + ipfs - https://github.com/ipfs/notes/issues/84 - this is a super rough way to show how to add ipfs support to a package manager through FUSE. it's not the best way, but it works!

* also, we <3 nix, and cool stuff coming in the future there :)

We have LOTS in development and store for the package manager communities. It's a very exciting time. Please join us at https://github.com/ipfs/ipfs and #ipfs and #gx on freenode IRC. And, if anyone wants to work on "Hypermodular Programming" with us, see https://github.com/jbenet/random-ideas/issues/27 and ping me :)

Have you tried using all of them and compared them for yourself?

git, hg, monotone, ..., all offer(ed) similar feature sets -- dvcs. yet they're very different tools.

haha, wouldn't this be so interesting?

wish wolfram was open source

Take a look at this: "Content Model, or Replication on IPFS" https://github.com/ipfs/faq/issues/47 -- it's a discussion on WHY we had to separate out the replication part. Check out also:

- ipfs-cluster - discussion on a tool we'll build to replicate archives to many nodes and have a RAID-like cluster configuration - https://github.com/ipfs/notes/issues/58

- ipfs-persistence-consortium - https://github.com/pipermerriam/ipfs-persistence-consortium

- pincoop - https://github.com/victorbjelkholm/pincoop

- http://filecoin.io (long term solution)

- more at https://news.ycombinator.com/item?id=11137313

Thanks for the feedback! We have answers to all these questions over at the many repos that we use to work on IPFS. Look around in:

- https://github.com/ipfs/faq/issues

- https://github.com/ipfs/notes/issues

- https://github.com/ipfs/apps/issues

The technical answers exist. Please look for them! I'd love to write personalized answers for every person that asks, but i better spend my time developing. Also you should look at the rest of the discussions too, not just my compressed answers. be part of the discussion, raise your concerns, and have them answered.

For sake of saving you time, some short / compressed answers:

- mutable data: look into ipns (in the ipfs paper or repos linked above) and look into CRDTs. CRDTs can layer cleanly on top of ipfs for distributed mutable state. if you haven't seen CRDTs: http://hal.upmc.fr/inria-00555588/document

- access controls / certain info remaining secret: data encryption and capabilities. you mention Tahoe-LAFS, yep!, we can (and will) build a cap system similar to it on top of raw ipfs objects (ideally actually collaborating with the excellent Tahoe-LAFS team directly). --- oh and many users want easy selective disclosure for encrypted multiparty records. we've not built this in yet because we have other priorities right now. if you'd like to contribute to this we'd love the help!

- "it will have to not only cover every existing HTTP usecase" -- we don't aim to cover _every_ HTTP facility, instead we can power a nicer model for distributed data, where you replicate datastructures directly. This looks a lot closer to the data models people use in apps today (think single page app models, {backbone, react, meteor, ...}. these are layered on top of APIs like REST, but can trivially layer over IPFS too)

- pub/sub: you didnt mention it, but it is necessary for fast (subsecond) updates to mutable data in the large (>millions of nodes). we're working on these designs now, and impls coming in a few weeks/months. (some simple non-scalable versions already proposed and implemented to get started, but the long term solutions coming later).

I ask you please take a deeper look at the comm channels of our community, and that you voice your concerns + questions in our FAQs and other relevant repos. That way we can improve our models and systems based on your input. And if you have time to help us implement some ideas, join us! :)

more links at this answer https://news.ycombinator.com/item?id=11137313

<3 this comment. would love to see the actual probability too. "vanishingly small" sounds about right :]

Thanks very much to Christian and John for writing a much needed detailed article :)

Some more links for people to check out:

## (upcoming) IPLD "merkleized JSON" format:

- improves upon our basic format to make it much more pleasant to build things on top of ipfs.

- JSON meets CBOR meets Merkle-linking

- mini-spec: https://github.com/ipfs/specs/blob/master/merkledag/ipld.md

## answers to some common questions i've read on this page:

- content model / replication: https://github.com/ipfs/faq/issues/47

- how resolution works: https://github.com/ipfs/faq/issues/48#issuecomment-152917088

- how IPNS / mutable linking works: https://github.com/ipfs/faq/issues/16

- this is a very poor answer, sorry, i'll write up a post or paper on it.

- for now if interested, see the QConf slides below, specifically slides ~110 to ~130 -- the DNS, IPRS, SFS/Mazieres linking, IPNS parts.

## These repos have interesting "lab notebook" style discussions:

- https://github.com/ipfs/notes/issues

- https://github.com/ipfs/apps/issues

## deep dive talk at stanford:

- video: https://www.youtube.com/watch?v=HUVmypx9HGI

- slides: lmk if you want them, i'll pdf them up

## talk at ethereum's devcon1 covering blockchain uses

- video: https://www.youtube.com/watch?v=ewpIi1y_KDc

- slides (interesting bits start at slide ~70): https://ipfs.io/ipfs/QmUgRq7QfmRbPw5kXqwSs1TRtPDBXMoDNiYwJQg...

## talk at qconf sf (similar to above)

- in this talk i discuss a bunch of datastructure stuff, including using IPFS for PKI, for arbitrary dns-like records, for name systems, for CRDTs, and so on.

- unfort video will be released in march: https://qconsf.com/video-schedule

- slides (intersting bits starts at slide 80): https://ipfs.io/ipfs/QmPpYmdSEKspjgXxVyGK9UMHV54fKZS8MwJjppg...

Science converges on truths, so similar (and even the same) ideas form in many minds, often at the same time. And fundamental ideas like these are often -- in hindsight -- revealed to be pretty old; you can trace their lineage and formation across papers, talks, and books. Many people in the field, with a good measure of "conceptual agility", will get this quickly and be able to discuss at length. Not to detract any credit from individual discoverers: they are absolutely critical for the quantized improvement of ideas; often a single person will move science forward in ways the rest of the world could not. And sometimes it is many people that do. A constructive memetic environment is good for both. (it is unfortunate that Credit is THE scientific currency today; it interferes negatively...)

No doubt colah's excellent blog post dives much deeper than this essay, but please note that this was wrtten for Edge.org's http://edge.org/annual-questions book, for a general audience, with very high level perspective, with very little space, and without the ability to get technical. Don't pit them against each other! They complement!

By the way, for one interesting example of the "conceptual agility" of scientists at the very edge, check out this interesting story about Feynman -- from a David Deutsch interview (with Sam Harris) -- in which Feynman derives months of Deutsch's work in a few minutes (listen for ~5 min):

https://www.youtube.com/watch?v=J21QuHrIqXg&t=6524 (1:48:44 - 1:52:40)

Scientists' minds are well primed to understand, derive, and formulate ideas -- they are super fertile memetic environments. And what's more-- they have the epistemic filters to annihilate untruths and converge on scientific knowledge (the best explanation not yet falsified). In math (inc. computer science), it's even better because we study and manipulate the objects themselves, not data from measurements of the things. The formulation, derivation, convergence, remixing, (and so on) of ideas is much, much faster.

Knowing both colah and davidad, I confidently assert they are both among the most brilliant young scientists alive. Humanity stands to gain much from the constructive interference of their minds. As readers and commenters, let's foster that.

love seeing papers like this: improve abstractions, implement, win.

true, but then you're not in a {censorship, attacker} free environment at all -- MITMing TLS _is_ violating the security expectations of today's websites. I wouldn't even check my non-work email.

i.e. all bets are off. there will always be pockets. but over time we could win even there, as the perf improvement will matter.

Full SQL is much harder to implement, JOINs done well (read: fast) are tricky, constraints and so on as well. NoSQL is easier because it punts on all the hard database problems and leaves it up to the application layer.

not yet, but we should! will email them, thanks for ptr!

It's more than that, take a look at the other comments in this thread, like https://news.ycombinator.com/item?id=10329236 and https://news.ycombinator.com/item?id=10329221

One interesting thing is you can have totally end-to-end encrypted applications, encrypt the application code and all the generated data.

Also, remember that the HTTP Web itself was a reimplementation of decades-old hypertext systems. Hypertext hails all the way back to Xanadu (60-80s!).

+1 Yeah, exactly! It's an exciting time to be working on all of this.

(IPFS author here)

It turns out that:

- (a) IPFS already works

- (b) IPFS layers over CCN extremely well AND generates demand FOR CCN. (i'm a big fan of CCN actually, and want to see it in more and more systems. But CCN has huge adoption problems. Look, we don't even have IPv6 fully deployed yet! So if all IPFS does is generate enough demand for CCN to be fully deployed, we've done a good job.)

- (c) all the routing problems are very real, but can be solved just fine. If you haven't looked deeper at how IPFS actually works instead of short summaries, you'll realize that the IPFS specs define the routing layer as entirely pluggable for this reason: we need to evolve to better and better schemes over time.

Instead of blindly saying "this cannot work", read more first. Understand the systems _goals_, decisions, and roadmap. Ask questions if you're unsure how something could possibly work. Lots of very smart people are working on IPFS and we're doing it because we see that it can indeed (and does) work. Blind negativity does not help CCN, and does not help anyone make better things.

(IPFS dev here) wow, that sucks. is this blocked in your corp firewall? Any chance of talking to the sysadmins? Blocking IPFS is like blocking HTTP :/ :/ :/

We may have to move our gateway back off our main domain, and have a "recommended blocker", so that people don't jump on this.

Anyway, one piece of good news: despite everyone's belief about UDP transports never going through corp firewalls, QUIC now handles MOST (>60%, close to 80% i believe) of "Google Chrome<-->Google sites" traffic across the world.

So, temp setbacks may be annoying, but in the end, once users install IPFS -- and once we have browser implementations, we can make all the traffic look like HTTP TLS flows, so blocking it will be very hard without also blocking regular HTTP TLS traffic.

IPFS is not _just_ a static system. See the comments above: https://news.ycombinator.com/item?id=10329221 -- note that git ant bitcoin are "just static systems" as much as ipfs :). It turns out all content is "static", it's just "static content" that is "dynamically generated" at times-- the point being that you can still do all the dynamic stuff just fine, and use IPFS as a transport for the data.

(IPFS dev here) +1 o/ NDN is awesome. If any NDN devs see this, please reach out. We'd love to work with NDN on things-- we think we can generate lots of demand for NDN

Thanks mark! gun is really cool! Would love to talk about it, actually. I'd love to make gun a transport for IPFS objects, and viceversa. send me an email!

Also, IPFS works today too, https://ipfs.io is served directly via IPFS :)

(IPFS Dev) Thanks! :) Yeah, we want to help sites scale without needing massive infrastructure funding.

Thanks! and please do-- find us on github.com/ipfs/ and #ipfs on irc.freenode.org :)

(IPFS Dev here) Maybe. I want to live in a web where,

(1) if a region's network uplink disconnected (like it was in Egypt) websites/webapps don't cease to work. People should be able to communicate + compute in these local networks without backbone access. yes this is possible.

(2) if i manage to make contact with you over a totally unorthodox data channel, like ham radio, or satellite, i should be able to _easily_ pipe my traffic and updates to you, and do so in the git-style of replication: offline first + bursts of condensed traffic.

this is not rocket science. (this is rocket science: https://www.youtube.com/watch?v=Pl3x71-kJGM :D) our problems are much easier to solve, and we must solve them.

(IPFS Dev) Duly noted! will do! :)

(IPFS Dev here) Sort of, actually. The gist is that in general yes, the network is growing and getting better. But, there's actually a growing divide between the hyper-developed cities and ... the rest of the world.

One example is this: if your bandwidth is too low, or your latency too high, you basically cannot use the Web as it is today. Native mobile apps do muuuch better, because you download them once and can use them mostly offline or with low network usage. The reason this is "getting worse" is more of a perception problem and an artifact of this fact: storage is getting cheaper faster than bandwidth is getting cheaper. Over time, our media usage (and thus perceived requirements) grows and grows, tons of websites today are _several megabytes!!_ and all the media we use keeps growing to fit our nicer screens. Also, as more and more people come on line, and their usage increases, the networks saturate. This causes the "perceived bandwidth" to decrease, meaning that the pipes (which are getting absolutely better) can handle a smaller percentage of our individual load and thus "feel worse".

Then, there's are other stupid problems, like the fact that many major websites/webapps will be totally useless if you go over certain latencies (particularly with wireless meshes, meaning lots of packet loss). This is because the servers' request timeouts are way too low. I was recently traveling through _Europe_ and in many places (trains on the countryside or small cities) the mobile (and sometimes wired even!) latency was so high that i could not browse the web. TLS handshakes wouldn't complete. HTTP servers would give up on me. It was terrible-- as a person accustomed to LTE + fiber ("Aaaaahhhhhh the data!!!") I couldn't believe just how stupid and bad of an experience we're giving our users out there. This was Europe-- now transport yourself to Bangladesh, or many places in rural India where people are beginning to be plugged into the internet. Think of places where access to Wikipedia, to Khan Academy, to the messenger services, could make huge life-changing differences for people. Having a bad web _there_ is blocking people from having the amazing powers of communication and computing that we all get to enjoy.

We must fix these problems. And we're going to fix them by improving the data model and the distribution protocols, not with optimistic policies.

(IPFS Dev) Sure!

We divide naming in two parts:

(1) Providing a long-term reliable mutable pointer. (no consensus needed) (2) Providing a long-term reliable short and human-readable identifier. (consensus needed)

Where "long-term reliable" means i can rely on it for decades for important businesses. I.e. nobody will just take it from me by a fluke of the protocol.

IPNS, the naming system of IPFS, separates these into two steps:

(1) First, it makes a cryptographic name-system (this is based on SFS -- by David Mazieres -- look it up, fantastic system and a prelude to the core design of IPFS, Gnunet, Freenet, Tahoe-LAFS and many other systems). This cryptographic name-system means a "name" is the hash of a public key ("eeew that's ugly"-- yes, hang on). That hash name can be updated only by the holder of a private key (how? via the DHT and other record distribution systems, more on that later). The important part is that it (a) does not require consensus at all, anybody can make names (it's just a key pair!), and (b) it can be updated really fast over DHT, Pub/sub (multicast) and other network distribution systems.

(2) Second, it delegates the human-readable naming to _other, existing_ name authorities (note that _stable global solutions_ to this problem require consensus). We don't want to have to make _our own_ naming authority, lots exist already: DNS, all the DNS alternate universes, and more recently in the cryptocurrency world: Namecoin, Onename, and even Ethereum is making one. So, _instead of adding one_, we just work with all of them, and integrate. You can bind an IPNS name (a public key path, like `/ipns/QmbBHw1Xx9pUpAbrVZUKTPL5Rsph5Q9GQhRvcWVBPFgGtC`) to a name in those authorities _once_, and never have to do it again. For example, with DNS you do this:

   1. setup a DNS TXT record like: dnslink=/ipns/QmbBHw1Xx9pUpAbrVZUKTPL5Rsph5Q9GQhRvcWVBPFgGtC
   2. continue using QmbBHw1Xx9pUpAbrVZUKTPL5Rsph5Q9GQhRvcWVBPFgGtC as usual.

You'd have something similar with Ethereum, Onename, Namecoin and so on: you just link to the IPNS name once. Now you can use your private key to update that name whenever without paying the cost of going on the consensus network. And So, resolving an IPFS url like:

   /ipns/ipfs.io 
   -> /ipns/QmbBHw1Xx9pUpAbrVZUKTPL5Rsph5Q9GQhRvcWVBPFgGtC
   -> /ipfs/QmcQBvKTP8R7p8DgLEtKuoeuz1BBbotGpmofEFBEYBfc97

(Note at time of this writing, /ipns/ipfs.io links directly from `/dns/ipfs.io -> /ipfs/QmcQBvKTP8R7p8DgLEtKuoeuz1BBbotGpmofEFBEYBfc97`, not through IPNS, as this is good enough to run a static website for now, and it makes it more robust as we experiment with lots of IPNS things).

One more thing: resolving names via local-names and paths (i.e. a web of trust, using either SDSI style naming, or SFS's much nicer path version) is entirely possible and averts the requirement of consensus for meaningful human names. This is really useful and cool, and we will experiment with it in the future. But in general, this doesn't (IMO) give you the ability to do "global long-term reliable" names, as "jbenet" might mean something different to different segments of the network, so i couldn't _print_ the words "yeah, just go to `/jbenet/cool-site`" in _paper_, because there would be no global consensus for `/jbenet` and i would like to make sure all my references are viewable by anyone across space and time.

Hope this helps!

(IPFS Dev) Yes yes yes yes! :)

This is very much what we're going for.

Glad you mention semantic web-- it tends to be a very tricky word with most hacker cultures, because everyone loves to hate on failed attempts :( :( -- but in fact SW (Linked Data!) is a super interesting model that has made Google and Facebook tons and tons of money (Knowledge Graph and Open Graph!). One interesting fallout of IPFS is that we can make Linked Data waaaaay more robust and faster, because you no longer need the _insane_ tons of little HTTP hits to query, retrieve content, retrieve other content + definitions, etc., on each request. With IPFS we can turn everything into a (merkle)dag and distribute it as a single bundle of objects. think Browserify/Webpack but for Linked Data. All with valid content addressing and signatures :)

And +1 to plan9 -- plan9 (in particular Fossil/Venti and 9p) have always been great inspirations for me, and their philosophy surfaces a lot in IPFS. From the fossil/venti (git) approach to content, to making everything mountable, to just using the path namespace for everything, etc. :)

(ps: woah, a message positive on BOTH Linked Data AND plan9! unexpected find!)