HTTP is obsolete. It's time for the Distributed Web (2015)
blog.neocities.org
blog.neocities.org
Imagine that you and some friends want to launch a small local business and need to host a website. Instead of paying to host it "up in the cloud", why not plug a few raspberry pi's into the walls at each of your houses? Between that and also seeding it from your laptops, the site should have decent coverage. Maybe you could also offer some discounts for loyal customers who choose to seed it. This site will be redundantly hosted in the location where it's most likely to be accessed.
If you're doing something more permanent, then you will want to make sure you have more stable hosts, and not just some peoples cell phones. Just like you do now. It's the same thing as now. If you want to host something, make sure there is at least one computer that is serving it on a decent internet connection.
The difference is that with distributed web technologies, there is a smooth continuum for scaling. You don't even need to assume there is an ISP to seed! All you need is LANs. But if you want to do big things, you can harness the power of thousands of peers all streaming something that they're into.
I use syncthing and resilio sync all the time, and it works great with just 3 devices.
To provide an argument that might fill this gap:
Most systems don't actually have a huge amount of data. Look at the data size and data growth of CRMs, special-purpose wikis, and so on: These are mostly smaller than 500 MB (excluding static content like images), and grow by less than 1MB even on a busy day. And that's the uncompressed size.
Also, most systems, despite being mutable, actually want (or need) an audit trail. So these are really append-only systems which merely have a "mutable look and feel" to the user.
I'm not following the logic of the argument here though, jumping from "X is good for Y" to "...don't actually have huge amount of data", perhaps you can elaborate?
With a merkelized append-only log (immutable DAG), there's always an audit trail. I agree with your point about "mutable look and feel", in a lot of use cases there's only a limited set of "writers" and updates happen infrequently.
Perhaps I should rephrase my previous comment, then, as "immutable systems are good for building mutable systems on top". Does that help to provide a better counter argument?
You can build mutable systems on top of immutable (append-only) systems. But is that a good idea? Yes, it is, for systems which don't have huge amounts of (non-static) data, and/or system which need an audit-trail anyway. And these are more systems than one may initially think.
Why would something like a chat or email app need to hang onto that much history?
Imagine a distributed "email" app that uses networks of mutually trusted peers to deliver encrypted messages ("emails") asynchronously. My device doesn't need to hang onto your emails indefinitely. It only needs to hang onto them until they've been received. This could be done via explicitly sending receipts, or probably in most cases by giving stuff simple expiration dates. The sender would have the most incentive to hang onto the original message until its been delivered.
How this scales in terms of MB and GB is hugely dependent on how your application is configured, how frequently new data is emerging, the limits set by peers for how much they're willing to share, etc. But text is pretty cheap. I can't imagine storing 3 GB of yours or someone else's text emails on your phone, short term or long term. The raspberry pi plugged into the wall at your house can has much more storage anyway ;)
I disagree that immutability is a negatively defining factor here re. data size or capabilities of the database.
If you look how many Big Data systems process data, you'll find that at the core of many, is an append-only log. For example: Kafka is a log (https://engineering.linkedin.com/distributed-systems/log-wha...), and looking at Apache Samza's architecture, we can see how a log is at the core of it (https://www.confluent.io/blog/turning-the-database-inside-ou...). In less Big Data orientated databases, there's always a log of operations (sometimes also called a transaction log or replication log) to keep the track of changes.
As with git, even if a version (commit) is immutable, it doesn't mean it's worth saving. Lots of times, you might make a temporary branch locally to do some work. Then you'll merge it and push the merged version upstream. Later you might check out a new copy from upstream, not caring that your temporary working branch isn't there.
User friendly versioning is a major challenge for dynamic, distributed applications. How do we gracefully bridge the gap between long term (distributed) memory and short term (local) memory? Each specific application has its own needs and tradeoffs.
And how do applications communicate about which versions are compatible with the applications' needs? About which versions are worth holding onto?
There are examples of the use cases you mention being built with decentralized technologies. [1][2]
[1] The various cryptocurrency wallets and exchanges
Applications like Tox or Matrix (which uses servers, but not necessarily "centralized" servers) are great examples of dynamic p2p applications.
Or for example applications that use statically distributed javascript to facilitate dynamic p2p communications. Stuff like together.js, gun.js, freedom.js, etc.
Syncthing and Resilio Sync are also wonderful examples, and Resilio Sync has amazing encryption features: you can give out seed-only links to your data. People who use these links won't have permission to decrypt the content. They will only have permission to echo it. That's a "raspberry pi plugged into the wall at the coffee shop" solution to private, mutable content distribution.
As for the shopping cart example, this is something that could be conducive to a more centralized approach, especially if your physical distribution model is centralized and your payment system is centralized (traditional banks). In that of case, you'd want to have a more direct connection with the physical distributor. If you want a direct, instantaneous connection with the shopping cart company's servers, then that's what you need.
But it's possible to have a situation where your product is not physical (like music or video), and you are using a decentralized currency (like bitcoin). There's absolutely no reason you couldn't facilitate that in a completely distributed way.
By the way, Bitcoin is a banking app... Have you ever used a browser based cryptocurrency wallet? Imagine a browser based cryptocurrency wallet that's hosted on IPFS. That's a pretty distributed banking app. If you want privacy too, use zcash or monero.
Setup, updates, maintenance, tech support, and uptime guarantees, just to name a few reasons that "the cloud" is better. A service like Wordpress.com or Wix beats the self-hosted Pi on all of these counts.
I interact with a lot of non-technical small business owners and am "that tech guy" in their minds. A question I'm hearing more and more frequently is _why even bother with a website when a Facebook page is much easier and they can see people interacting with it._
Their reasons are not all that different from why many tech savvy HN readers are using a Mac instead of Linux: convenience; less shit to worry about.
Hosting anything on a Pi plugged into the wall goes in the exact opposite direction from what these people want. The centralized services are winning because they pay attention to what the market wants, they build it, and they make it easy to sign up.
If my roommate can plug a Roku into the TV, and knows how to use Ableton Live and Squarespace, there's absolutely no reason he couldn't use something like that.
And there's no reason that those non-technical people couldn't continue to pay you for helping them use stuff like that.
And there's no reason that people can't continue to use stuff like Facebook. But I have a feeling that people are going to be over that way of doing things by the time the next two decades are over.
For the price of hardware we're talking about, it'd just get replaced.
If their home network is down, hopefully their office network isn't, or their business partners' networks aren't.
Also, these kinds of applications are practically begging for mesh networks. So what it means for a home network to be "down" could change a lot in the coming years.
The Internet is breaking all the time anyway. Every day at least one of the bigger / more important sites I visit has a temporary problem with something. Three days a week HN keeps returning CloudFlare errors to me. Even Facebook has some issues that break it every other week. The world isn't ending because of this, and it isn't going to end because the site I co-host with my other friend is down for the night.
As you grow to the point where close-to-perfect reliability matters, you'll be able to afford to get someone do handle the hosting for you, just like you do today.
The bottleneck I see is bandwidth and government regulation of the electromagnetic spectrum. If Amit Pai gets his way then what you are talking about will be much more difficult.
Seems like the future to me, it's just not in any big tech company's best interest right now as they fight over data centers.
What if publishing to your blog was as easy as installing a Chrome web extension that now travels with you? Suddenly every computer you have that runs the extension can pin a local copy of the site, and your visitors help out by virtue of the protocol. This doesn't exist of course, but thanks to ipfs-js, it could, very easily. There are obvious drawbacks to this approach of course, but my point is that the "Hello World" of this kind of app could easily be much lower bar than setting up a Raspberry Pi.
I abandoned my Facebook account many years ago and refuse to set up a new one. As a result, it is difficult for my children to interact with me. They post their lives on Facebook, but they don't open that up to the public. So only specific Facebook accounts get to see what they share. That excludes me.
I have to open myself to Facebook to see their content. Alternatively, they could set up non-Facebook websites and "blog" their lives there. But that is harder than just using Facebook, and gets really hard if you want to keep it viewable only by select people. And then I would have to set up an account on their server to be able to see their content. Multiply that by all the people I would have to set up an account with and it's obviously unworkable pretty quickly.
I still hate Facebook, and refuse to set up an account with them. But I recognize that the alternatives are not pretty.
Edit: typos
The solution needs to allow identities which are "register once, use anywhere" across the web, and portable so that you can migrate to a different identity host/provider/implementation without losing all your accounts. Ideally these identities would also allow you to reveal as much or as little information about yourself as you want, and not force you to reveal some unique property which is correlatable between colluding sites.
Obviously that's not an easy thing to create, and may actually be harder than creating a viable rival to Facebook, but if we don't solve the web ID problem, then governments and corporations are going to "solve" it force us, and we'll all be the worse as a result.
It's a fascinating idea to have some means of identity authentication where the party seeking to authenticate your identity doesn't have enough identity information to connect the dots, and the party providing authentication doesn't even know about the party seeking authentication.
Setup, updates, maintenance, and tech support is however a thing, but there is always the fine print. The cloud generally do not maintain and update a website, and attacks generally succeed today by targeting poorly updated websites rather than servers. That leaves setup and teach support, two things which the website developer will often provide while they create and maintain the website.
That assumes that all users are morons. If they want to pay anything to use else's hardware and energy then they only need to pick up their wallet and offer real cash. Offering another Dogecoin competitor to convince people to give away their energy bill and hardware is just a fancy way to officially assert that the people who fall for these gimmicks are complete morons who give away their resources in exchange for glorified arcade tokens.
Distributed computing certainly has it's role, but the added complexity of cache invalidation, versioning, and content synchronization are undeniable.
I hear you on the added complexity of it all, and I think that's why its taken so long for this stuff to be developed. It's been stewing for a long time. LANs have been around since before the internet, but a lot of stuff that's relatively easy on the internet is much, much harder in distributed and mesh networks. I think we're currently witnessing an epiphany.
Having the ability to recognize the same resource, and to request it, from multiple, disparate devices in just about any network setting is a totally different level.
And on a human level, I believe it could foster healthier social interactions around the sharing of digital content. When content is in its infancy, these kinds of applications create incentives, however slight they may be, to share it in closer physical proximity to people. It gives us back the awesome joy of the LAN party. Facebook says "you need to share your content via ISP and it must be stored on our servers at all times." Whereas distributed network applications invite us to imagine more humane, local, trusted environments for sharing digital content. If you want to share old family videos, get together for dinner, watch them on the TV and then share them privately over the WIFI. If you're in a band, make arrangements with local cafes to host your album. People who come for coffee will have the best bandwidth on it. If you're an artist, host your portfolio on your phone. Share it with people like you'd share a business card. If they like it, they may choose to help seed it, even without your asking. We're talking about the internet for LANs.
This stuff has the potential to refresh the art of digital collection and curation, something that's been co-opted by centralized content providers.
That's, flatly, a lie. There's a whole passage in the middle about how needing servers is a weakness of HTTP that this gets beyond.
Instead, we could put Raspberry Pis in our walls? We can do that now.
In a distributed web you need seeds. Whether they are called "servers", "peers", "devices", or "clients" is dependent on implementation and semantics. For the purpose of my comments I chose to call dedicated seeds "servers", because it made sense to me.
And yeah, we can do that now! I've even used an old cell phone with a busted touchscreen as a low power syncthing device. Mostly it was meant to relay notes between my phone and laptop if one or the other was sleeping or dead. It worked great. Eventually, I stopped depending on it, because I have a headless machine running now (a "server") that does the same thing.
All I had to do to migrate was share my syncthing directories with the server. It was a breeze. I didn't have to worry about configuring addresses or anything.
Also, the server is useful as a relay, but if my phone and laptop are both on, say if I'm working at the library or something, they just communicate directly. So I don't depend on the server in the same way.
But the Android interface kinda stinks, so when I wanted to use it with "non-tech" people, I decided to just use Resilio instead. It's quite a bit more refined. I wish it was open source, but you can't have everything.
If it would work, and the cost benefit ratio were there, people would adopt it quickly. That's what happens with just about everything else.
I think they don't care much about the underlying technology, but they will notice when some apps work faster and without a mobile data connection when others don't.
Even with static websites you usually need to have a web server you are able to connect to, or have to go out of your way to add a Webworker that makes is offline-capable. There a single address that is served via an IPFS gateway behaves better with less additional tooling.
So will rapidly increasing data allowances and cellular coverage. Plenty of European countries have effectively unlimited data packages and effectively complete network coverage.
Crucially, whatever gaps there are in this are very likely to get filled through already underway progress much faster than a distributed web on mesh networks will get to a usable stage.
The status quo however is colleagues of mine discussing which of the main carriers to choose to get decent 3G/LTE coverage in Berlin(!), after all these years of progrrss, so I think it's worth to consider other options. Rural Germany is even worse. My data caps are also about the same (less than double) as they have been 5 years ago.
Even if it won't take over as the dominant technology, it might create enough pressure on the carriers to act.
Content-addressing doesn't alleviate the problem 100%, since content can still fall off the network - but it improves the structure of the network in a way that makes it tremendously easier to keep content around. It's not up the original source of the content (owner of the domain name) to keep the content around - anyone can help out by keeping a copy.
My colleague Matt addressed this beautifully in a recent talk at the NSDR Symposium: https://archive.org/download/ndsr-dc-2017/04_Speaker_3_Matt_...
You unknowingly make my point. I have nothing against people working on new technologies until they work. That's a strawman on your part.
But, this isn't a case of working on something until it works. The headline of this blog is, "HTTP is obsolete. It's time for the Distributed Web." IPFS is not ready to replace http, and it won't be until the cost vs. benefit ratio works out for enough people.
Great point! Just like:
* Betamax * HD DVD * Minidisc * Hoverboards * IPV6 * DNSSEC * PGP & PKI * Linux desktops * Dvorak keyboards * The metric system * Decimal time * [flavour-of-the-month programming language] * [flavour-of-the-month database] * [flavour-of-the-month cypher] * ...
The factors that influence the proliferation of a technology are wildly divergent from the criteria 'works well, cost/benefit'. I'm not even sure those are weakly correlated proxy indicators of technology uptake.
"Better" has to actually "be better ENOUGH" to warrant all of the retooling of existing systems. I've got plenty of clients who would happily run Windows 2003 ("it's paid for") if it weren't for changing standards that aren't compatible (newer TLS, Exchange, etc) and security breaches. They only upgrade because they have to. "E-mail is e-mail" to them.
But if you sell them some magical new technology that promises to meet new features, like tons of data analysis tools and easily graphs and charts in a new version of CRM, they'll happily upgrade.
It's pretty visible in tech that it's not actually the only (or main) reason, especially when you see companies continuously switching from one crappy tool to another. Tech is a fashion-driven industry; companies use what is hot and/or what everyone else is using. Both of those create a positive feedback loop that amplifies brief spikes in popularity (easily exploitable through marketing) beyond any reasonable proportion.
The worst thing is, though, that it kind of makes sense from the POV of management. The more popular something is, the less risk there is in using it, especially when the decisionmaker doesn't have enough knowledge to evaluate the options. Also, the more mainstream a given technology is, the cheaper and easier to replace programmers.
While I agree with your sentiment, this one is a bad example.
I grew up with the metric system, as did the vast majority of the world. I have an intuition for "meter", "kilograms", "seconds", and so on.
I need to convert to cumbersome stuff like "miles", "inches" or "pounds" only when reading articles written by, you know, inhabitants of that strange, large country over there.
[1] The history is actually more complicated, but let's not get into that.
That doesn't seem very logical at all, that's entirely subjective. I'm fairly certain a top sprinter would easily be able to sprint much further than my (admittedly) unfit self before running out of breath.
The point isn't that the measurement is precise, the point is that it's useful. The unit has an intuitive and tangible meaning in the real world that let's people ballpark. This doesn't mean we should start doing precision work in furlongs but demanding that everyone switch away from measures that are still useful is silly. As long as the measurements are standardized using metric units who cares that you have a funny name for 201.168m?
Metric is great for general use simply because of it's multipliers: (...) 1G = 1,000M = 1,000,000K = 1,000,000,000 = 1,000,000,000,000m (...)
And also the simple way many units are related as well, like 1L of water having 1kg of mass (yes, with a certain temperature, pressure, yada yada yada)
You do care if you frequently have to convert between all those funny units.
Another example of a funny name is two-by-four, which - for some typically American reason - is understood not to actually be two inches by four inches...
> The metric system
Really? You know that the whole world is on it, right? And that it makes far more sense than whatever nonsense someone came up with before.
(Here's my new conspiracy theory: lack of adoption of IPv6 is caused by SaaS companies colluding to keep people and companies from being able to trivially self-host stuff.)
It just seems silly to talk about how HTTP is unreliable because your severs might go down, when the alternative "serverless" architecture you're hyping doesn't work either. I'm totally on board with the aims of IPFS and hope they accomplish all the things they're trying to do, but to say HTTP is obsolete when HTTP works and IPFS doesn't (yet) is just a little too much...
The post makes the case for why IPFS should replace HTTP -- has nothing to do with how well the technology works right now in its infancy.
For a truly distributed network we should look into ways to make internet work more like a mesh.
Back in university, all the student dorms were connected to the same internal campus network, back then internet was slow but you could still share files blazingly fast with all the other students on the network (using DC++ at that time). While this wasn't exactly a pure mesh either it shows that local data works and beats internet many times, even with just a few thousand clients. Then DC++ was mostly focused on pirated content but with a more human friendly solution, like IPNS, it's not unimaginable that average Joe neighbor with one click can create a local mirror of the whole Wikipedia for you.
Combine this with the recent work on PubSub [1] and CRDTs [2] and you can make many applications work locally, that are otherwise annoyingly strongly coupled to internet services (thing Etherpad, Google Docs, Skype, Github, etc.)
> For a truly distributed network we should look into ways to make internet work more like a mesh.
Yes! We'll be putting more work into the network stack (libp2p) in the coming months. IPFS itself has done a ton to rework how content is defined and moved around, and libp2p will do the same for the network connections underneath. Think overlay networks, cryptokey routing, packet switching.
It's the same dilemma that Tor exit node operators face. You are essentially loaning your IP out to the masses. Since this is what law enforcement uses to associate internet activity with individuals, there's a high risk that someone will eventually end up "borrowing" your IP to send something illegal.
This is a very business/customer-centric perspective. What about resistance to censorship, ease of sharing without relying on central third-parties, resistance to linkrot of non-commercial content or even content from a business that went under, etc?
I don't think Freenet and IPFS mutually exclude each other, though. Freenet is a specialized tool with low usability. IPFS aims to be a core, widespread infrastructural protocol with high usability. In this way, I think they can co-exist and be complementary. The existence of a harder to use and more anonymous tool doesn't negate the usefulness of having a more widely used tool with lesser guarantees.
> IPFS doesn’t require every node to store all of the content that has ever been published to IPFS. Instead, you choose what data you want to help persist.
Freenet has a different set of design goals and tradeoffs. IPFS is much close to Bittorrent which AFAICT has never really waned in popularity.
They also differ in the way routing works. On Freenet you ask a (mostly) random neighbor whether they have a file with the hash you want. If they don't have it, they ask another (mostly) random neighbor. This can go on for a while, until it either finds the content or hits a maximum number of hops, in which case it backtracks. The only point of these rube-goldberg shenanigans is anonymity. Since IPFS is more concerned about performance it flips this on its head: instead of blindly asking nodes for content, it carefully keeps track of peers who advertise what they're looking for; aside from being much more efficient, this also allows you to choose not to do business with leechers, like in bittorrent.
Maybe IPFS is a reinvention of a past technology, but certainly not Freenet. (Does anyone know of something closer?)
Freenet paper: http://www.cs.cornell.edu/courses/cs414/2003sp/papers/freene...
IPFS paper: https://github.com/ipfs/ipfs/blob/master/papers/ipfs-cap2pfs...
Actually there isn't that much of a difference here. Freenet manifests are analogous to git trees; knowing the chk of a manifest file gets you to the metadata that identifies all the files under the tree. It's all immutable.
There are some noteworthy differences though. One that stands out in particular is that freenode breaks large (>32kB) files into chunks. Everything is encrypted too. So finding a file you want is not quite as simple as taking the hash of the plain, unencrypted file.
Either way, you can easily build a git-like hierarchy of immutable content (and history) on Freenet, and this is more or less what happens under the hood anyway with manifests and splitfiles.
As a slight deviation from the norm, Freenet also can also address (signed) content by its public key rather than content hash. This is one way to enable mutable data; not entirely unlike heads. They can still link to immutable content hash keys.
> They also differ in the way routing works. On Freenet you ask a (mostly) random neighbor whether they have a file with the hash you want. If they don't have it, they ask another (mostly) random neighbor. This can go on for a while, until it either finds the content or hits a maximum number of hops, in which case it backtracks. The only point of these rube-goldberg shenanigans is anonymity.
It's worth pointing out that Freenet does have a simple but powerful routing system. Each network node has a virtual location, in key space. Requests are routed towards the nodes that are most close to the requested key. With careful selection of peers, the network topology can make for very efficient routing. E.g. one could have a small number of nodes "far apart" in key space, to faciliate routing towards far-away keys. Then you have a larger number of relatively close nodes, so that when a request comes in "your general direction" from a far-away node, you're likely to have the right peer to route to.
It is true that some randomisation helps with anonymity.
EDIT:
I'll add that the flipside of a network that essentially enables leeching is that it's also good for retention of popular (or "popular") items. Requested data is cached en route, so it's sort of automatic load balancing. Soon enough popular resources are likely to be held by whoever is nearby. People won't need to manually pin the content, and it's hard to directly DoS those who share the content you're "after." Asking for it just makes it more available.
I think this is important to consider if we're discussing reliability of distributed networks.
I'm not saying what the implications are -- for they can be good or bad.
It looks like I will also have to learn more about IPFS's routing. Clearly Freenet's has some merits, and I would hope that IPFS is strictly faster since it sacrifices privacy, but I don't understand it well enough to make sense of how it scales.
Right, but there in lies the biggest issue that holds these systems back. Do you:
- Replicate and cache data freely between nodes, and by doing so open up scenarios where unpleasant content is stored on and served from people's nodes without their consent OR
- Limit replication and storage to elective manual choices made by the user and/or recent data they have explicitly accessed, and in doing so severely compromise the ability of your system to retain and serve data as that data ages.
IPFS in it's current state is prone to as much if not more bit rot than the web as a whole - When nodes drop offline the content they have pinned is unlikely to be present on any other nodes unless the original host party has explicitly replicated to other nodes they also control and pin content on.
The only solution IPFS has for this currently is manual, elective pinning of content by other network participants. Realistically, if your replication and robustness scheme depends on manual user intervention it's not going to find wide adoption.
All of this is fine, IPFS still has usage scenarios it meets well when operating in the state it's currently in. But as far as some ideas being bandied about on how it's producing a censorship-resistant, bit-rot resistant persistent storage infrastructure that might replace HTTP... Nope, not unless a novel solution emerges to this specific problem.
- Because these centralized walled gardens were DOA; people just didn't realized it yet. Personally, I never saw any of the currently popular social network platforms as offering anything significant over the AOL of yesteryear.
- Now that its been shown Facebook and Google actively monitor and restrict content that doesn't align with their narratives, folks are becoming wary of centralization.
I think HTTP/websockets is very good for these things. Static data is one thing, dynamic is a whole other story. It seems IPFS is just a new distributed way to archive data. So what? It doesn't help serve something like FB over a distributed network does it?
And to what extent could some sort of "protocol" vulnerability stop these networks from being "uncensorable". Are they truly resistant to censorship, or could they be effectively shut down somehow? Wouldn't DDOS attacks cripple these? I mean, that's a crucial flaw, right, you just have to look up all nodes for an piece of content and constantly flood them with DDOS traffic and then, hey, you've censored the network, right?
Take look at OrbitDB (https://github.com/orbitdb/orbit-db) - "Distributed peer-to-peer database for the decentralized web" or their blog post "Decentralized Real-Time Collaborative Documents - Conflict-free editing in the browser using js-ipfs and CRDTs" (https://blog.ipfs.io/30-js-ipfs-crdts.md).
And all that works in the browser without running a local IPFS in the background. That's pretty amazing imo.
Leaving aside use cases like credit card information, there are a lot of user information that is illegal to share unless the user explicitly consents. In the EU you can't even share your access logs by default.
And how do you handle authentication? Passwords? how do you avoid user enumeration, the collection of user email and info?
Distributed filesystems and CDN in general are great, but let's use them for things that do not actually need a single bit of security, please.
The notion that distributed filesystems are inherently, or can't be, secure is way off. I would argue that with these technologies, such as IPFS, they can be more secure.
The use cases are not only "open databases" (by which I assume you mean open to public), private databases and data sets can be achieved just as well. Just because it's "distributed" doesn't mean it can't be private or access controlled.
Agreed on the comment re. "...illegal to share unless the user explicitly consents" and I believe this will turn out better in the trustless, distributed web, eventually. Our whole current approach is based on the client-server paradigm forcing us to put every user and their data into one massive centralized database. But we can change the model here. Instead, how about you owning your data(base) and controlling who gets to access it? "Allow Facebook to read your social graph?" "Oh, no? How about another social network app?". As a user, I would want to have that choice.
That bridges to your next point on authentication, which can be done on the protocol level with authenticated data structures. You can define who can read/write to a database by using public key signing/verification. It could be just you, or it could be a set of keys. One good example of this is Secure Scuttlebut (http://scuttlebot.io/). I highly recommend to take a look an understanding the data structures underneath.
https://github.com/ssbc/patchwork >You have to follow somebody to get messages from them, so you won't get spammed.
Doesn't that make it completely pointless because updates are still centralised? It merely shifted trusting a single provider to trusting each user which is not a scalable solution. The value add is so low you might as well just use IPNS and make people subscribe to IPNS addresses.
The problem is limiting read access. Having a globally distributed db means that anyone can get a copy.
You can use ipfs for public data, and to store private encrypted data (with caveats: make sure you change the encryption key/nonce for every data change).
There is no way to modify private data depending on anonymous access without things like omomorphic encryption, and the whole system is completely void of any form of forward secrecy.
As someone who works with encryption and security, I can not recommend storing anything private on distributed systems. Leaks way too much data, and there are too many caveats. You can have securely designed applications, but I see no way to safely port common websites completely on distributed infrastructure without leaking a lot of data that today is supposed to be kept secret.
Full p2p applications can't offload computation, computations have to happen on your computer and for computations you need the entire dataset. This is fine for messaging and static files.
Federation is a far better idea. You get the benefits of centralisation and decentralisation.
1. The availability depends on the number of peers like BitTorrent? If so, and if no seed is available, how does one access the content, esp in the context of an intranet?
2. Any change to how we run infrastructure except not serving HTTP?
What if a "universal basic income" meant "get paid for sharing your excess disk space"? This could even be made transparent similar to OS page caches, and then everyone with a computer + internet connection would be a participant.
There's a concept called pinning which keeps files available in a local share. You can pay other people to keep it pinned for a long amount of time, and they usually charge for the size of the file.
https://steemit.com/video/@heimindanger/introducing-dtube-a-...
Anyone can help out by keeping a copy. With location-addressing additional copies of the content aren't just largely hidden, they also get into a weird mode of competition with the original URL. With content-addressing, additional copies instead forge stronger resilience.
No, the original source is still the one solely responsible, unless there's some agreement between parties to cache content for them.
Anyone can help, yes, but it's not their responsibility to keep content online unless there's some agreement between the original content source and other nodes, which is of course not the case by default.
My colleague Matt addressed this beautifully in a recent talk at the NSDR Symposium: https://archive.org/download/ndsr-dc-2017/04_Speaker_3_Matt_...
// edited to fix URL
HTTP is ubiquitous, and no purism or idealistic superiority of some other protocol is going to sway everybody to the New Hot Thing. This is not a knock on IPFS but rather a recognition of reality: you're going to have to work within the current system to supplant it. And that means not only tolerating the old thing while pushing for the advantages of the new thing, but accepting that absorption of the dynamics of the new thing within the old thing represents a victory for the new thing, even if it doesn't get the named recognition.
Maybe we're ready to evolve beyond the limitations of HTTP (and HTTP/2, which I see as a viable and feasible, if not short-sighted improvement to HTTP). How are you going to get Google, Facebook, Amazon, and everyone else to go along with you? If you offer the benefits as compatible add-ons to the existing norms, you will succeed. If you demand that we fully jettison HTTP to achieve something better, methinks you will have an insurmountably hard time.
Fun fact: when URLs were being defined, some people expressed the opinion that location-addressing is a huge mistake to begin with. Almost all of the links and references in the respective mailing list archives are now: broken.
Currently aiming to release a Go implementation at some point after canning a C implementation earlier this year, where p2p is instead accessed via CONNECT proxy.
Is there any interest in this project?
Scheduled republication is my best answer so far.
If you promised to sign and republish the same file every day with a new timestamp, then people would know when they had the latest, and when they didn't... would just have to wonder if you fell off the earth, which we sort of do already with all those abandoned free software projects online.
Republication may be cost prohibitive for large files, so instead you could republish a metadata file that pointed to the latest hash as of the metadata file's publication time.
For the "hit by a bus" problem (or for a server doing this automatically, the "hit by a comet" problem?), it'd be nice to include a dead man's switch from a third party, where they can publish a "FINAL -- EXPECT NO MORE UPDATES"...
But that that point you're trusting a third party. If you're willing to trust a third party this is far easier. So that might be what we'd end up with... something like DNS providers, but they're suddenly managing indexes and metadata for hosted files? I don't know...
(Also, this has probably been worked out already by smarter people than me, I haven't looked at IPFS much, this was just a back of the napkin guess.)
You can use it as the base for CRDT structures: https://ipfs.io/blog/30-js-ipfs-crdts.md
Also: https://hashbase.io/
https://tokeneconomy.co/the-analysis-filecoin-doesnt-want-yo...
(With apologies to Winston Churchill.)
Combine IPFS and crypto currencies for payments and we have a new distribution standard. Then one could have distributed movie sites like Netflix and YouTube where you pay for royalties to legally file share the content.
But I suggest that internet is 'centralized' not due to any technological function, but due to the nature of information, particularly the economics and business of information.
The 'protocol' used to pass information from A-B, were it changed to something better, would not yield what the author is suggesting. Google 'would still have all our stuff' - I believe.
When 'individuals' can 'run services' from a variety of physical locales, with robustness - we will see more decentralization, I think.
I believe this will happen probably by accident, gradually, as 'more tech' creeps into our homes, one day, most people will have enough 'gear' to run some kind of service from home/small office - and other places.
Finally - despite the obvious failings of HTTP ... incumbency is what it is. I'll be we're stuck with it for a very, very long time.
If a powerful enough entity decided to change that - like Google AND Amazon together, or the US Government, or the Chinese Government ... that could change. Funny enough I think it's China that's best positioned to do it. They have the wherewithal and the tech momentum, they could do it by 'fiat' and in 10 years implement some kind of 'new, better network' that we'd all eventually move to.
Hey - America is still using 'swipe cards' and doesn't even use 'smart cards' though they've been in use for 40 years around the world :)
One of the things about the net, is that it's transactionally open.
There's no inherent identity/or security, it was grafted on with SSL - 'kind of'.
I suggest it might have been better if identity were required to even make connections, to avoid many kinds of attacks. Hopefully, it could be done in an unbureaucratic manner, also wherein 'identities' can remain de-facto anonymous.
My main point was that 'http' is kind of old, but we're stuck with it unless a 'major power' does something about it.
So yes, it's obsolete.
That's a sophisticated response. Though you might care to decide which meaning of "sophisticated" you think I'm referencing here.
(There are, in fact, words with widely varying, and even opposed, definitions. Arguing that someone means one that they clearly don't, in such a case, would be exceedingly uncharitable.)
This is you choosing to ignore the first definition. And it's NBD.
I do understand how language works. You can choose to focus on one definition and ignore another, as is clear by this thread.
Except, you're gonna have to have servers unless you've got the entire web backed up on everyone's computer. Otherwise, you don't and can't know how many copies of a page or other file are out there. But who's going to pay for servers to retain random peoples' and companies' web detritus? This whole project exists because that's not feasible in the long term...
It's not "crazy", it's pure, thoughtless hype.
>I am sure most people could afford that to manage their interneting
And before the spec has even seen real adoption, we've already seen it centralize into a few major providers.
A bit tongue-in-cheek, but it's a real issue. HTTP isn't the reason things are centralizing so much as economies of scale and convenience are. I see nothing about IPFS that fundamentally changes that, and think we'd likely see similar centralization over time.
Git, one of the inspirational technologies, is in theory distributed as well and in practice hyper-centralized to only a few major providers.
I'd much rather host content from my home (and, living in a very sunny Australian city, power that with PV and battery storage at a low amortised cost) - but I can't, because the network to support that isn't there.
I get about 7Mb/s down and and 1Mb/s up - my link is highly asymmetric. When I finally get off ADSL and on to the new National Broadband Network, that'll still be asymmetric.
I can see why networks are built that way, given the current centralisation of infrastructure, but the build also reinforces centralisation.
Think back to 20 years ago when most business network connections, even for small business, were symmetric. Hosting stuff on-site (email servers, web servers, ...) was far more common.
The kinds of tools that would make distributing those aspects of github are precisely what this article is advocating for!
Git repositories are replicated all over.
My laptop has mirrors of all my work's projects and many open source projects.
Imagine how many secure mirrors of, say, the React repository is out there. GitHub is basically just a conveniently located copy.
That's real and tangible decentralization. It's a magical property of the DVCS architecture that it's decentralized even when it's centralized, so to speak.
I agree that there are issues with central hubs though. Maybe the most significant one is that organizational structures and rules are defined in a centralized way on GitHub.
If you look at blockchains as another kind of DVCS that's totally focused on authority, legitimacy, and security, then it seems pretty likely that we'll end up using those to control commits and releases.
In fact the “web” is consolidating onto those who have the capital to do servers at scale.
In some ways this is good as it makes computing power more accessible to the masses with good ideas but on the other hand it puts the power of what happens with that business in the hands of far fewer people.
You mean Filecoin?
Anyway, I'm also a skeptic about this model but I do think there is a sliver of chance that it may work. There ALWAYS is a sliver of chance that something crazy may work. That's how it's always been.
IBM laughed when personal computer vendors and OS creators thought that they will put computers on everyone's desk, and I'm pretty sure if I was back then I would have thought the same.
Also, before criticizing some technology it would help to actually understand how the technology actually works. As far as I know, IPFS is working on all the problems you mentioned. Now whether they will succeed or not is a whole different issue, but it's not such a trivially obvious thing that one could easily say that it's a "thoughtless hype".
Checkout Freenet[0]. And while trying to maintain anonymity makes the problem even more difficult, there are fundamental problems in Freenet that make it basically unusable (mostly around cache coherency and latency). Freenet has been around since Ian Clarke's paper about distributed information storage and retrieval systems in 1999. They haven't managed to fix these fundamental problems in nearly 20 years of trying. I see absolutely no discussion of the same problems in IPFS (though abandoning anonymity is a good start).
It's one thing to say, "Hey distributed file system -- awesome". Then you can build all the easy bits and say, "Well, maybe cache coherency and latency won't be a big problem". But now look at what IPFS has to say about cache coherency on their wiki [1]. There is nothing at all that identifies or addresses the problems they will run into -- just a definition of the term and some links to random resources.
It's all well and good to say, "Eventual consistency", but what about guarantees of consistency? If I'm a vendor and I have a 1 day special offer, can I get a guarantee that caches will be consistent before my special offer is over? How do you deal with network partitions? Etc, etc, etc.
Before you start calling HTTP "obsolete", how about solving these kinds of problems? I have absolutely no problem with projects like these. They are awesome and I encourage the authors to keep working towards solving hard problems like the above. But announcing your solution before you've even realised that the problem is hard is pretty much the epitome of naivety.
[0] - https://freenetproject.org/
[1] - https://ipfs.io/ipfs/QmXoypizjW3WknFiJnKLwHCnL72vedxjQkDDP1m...
Second, people tried "sharing economy" startups back in the web1.0 era when everything went down crashing. But in 2017 we have Uber.
The freenet project doesn't change my argument at all because like I said, I'm not saying IPFS will succeed. I'm saying there's always a chance because the world is constantly changing. If you're lucky, you're at the right place at the right time building the right thing. If you're not, you fail.
In 1999 this wouldn't have worked of course, and that's my point. Successful projects succeed not just because of the product but also because of luck, timing, etc. There are so many new powerful technologies coming out nowadays, not to mention the societal change.
This is definitely a different world than what it was in 1999 and I'm saying just because it didn't work in 1999 doesn't mean it won't work in 2017.
What a well thought out argument, thanks so much for your insight.
- The content-addressing aspect makes it perfect for distributing commonly used libraries.
- We already cache all this content locally. What a waste! Why do I have to fetch jQuery from fricking California when it's sitting on my girlfriend's phone in the other room?
- This extends beyond the web: think about the benefits (both in security, practicality, and performance) of content addressing introduced into package managers (take it one step further even: combine this idea with the new move towards reproducible builds (https://reproducible-builds.org) and package managers like guix and nix and things get really interesting).
- It's actually easier to use for the average person. If you don't think this is the case I propose a simple experiment: download the beaker browser and set up a simple static site. I recently did this. It really is one-click hosting! Considering how complicated web hosting is to the average person (ever try to walk a friend through setting up a website? not. fun.) -- people would love to be able to set up personal websites this easily... and for free?
- As others have mentioned, there are many solutions being worked on for the mirroring of data (Filecoin etc).
- For websites that are visited regularly, this is not an issue -- all content is cached temporarily. It suddenly becomes basically free to serve an audience of millions... again: with one click.
- If history serves as precedent, if it does fail it would be in spite of being an objectively superior, practical solution. Getting a critical mass of people on this thing is the hardest problem to figure out. -- I suspect package management, academic data are the best place to start, then one-click personal hosting -- not even think about "apps" for now.
- Didn't you just read the web is about to go permanent? Do you really want to be archived for all history as one more nay-sayer? ;)
Because your girlfriend probably values her battery life and data usage? I doubt we'll ever see phones hosting any ipfs content for the reasons.
(and again: content-addressing could drastically reduce data usage anyway.)
Battery usage is definitely noticable, but my phone has a good battery life overall, and IPFS on the phone is a priority for me.
Data usage would be a problem, but I use it only when connected to a portable WIFI hotspot I am carrying with me.
[0]: https://play.google.com/store/apps/details?id=com.hobofan.ip...
[1]: https://play.google.com/store/apps/details?id=org.ligi.ipfsd...
Sorry but IPFS is interplanetary in the same way a Boeing 747 is capable of orbital flight.
Last I checked IPFS will not tolerate minute long latencies and requires a bandwidth above several kilobits per second which would immediately disqualify it for anything farther than the moon.
And I'm not sure it would work on the moon since that is a 2 second latency and I had issues with it when I used it on a mobile phone network with 800ms latency.
>I recently did this. It really is one-click hosting!
Except it isn't hosted unless atleast one person keeps a copy online, otherwise it goes offline or you pay money to some hoster or filecoin (not that I think that filecoin isn't a huge scam at this point)
>- Didn't you just read the web is about to go permanent? Do you really want to be archived for all history as one more nay-sayer? ;)
Since the number of people interested in the content of this page is declining with every decade passing, I think I'll make a bet it'll be no longer available on an IPFS after a mere two decades.
And the vacuum tubes in my Colossus might overheat at that rate too! -- Damn, you're right, we're just not smart enough to solve those problems.
---
edit:
> Does sarcasm prove your point?
Fair enough, sarcastic Parthian shot removed. I get overexcited sometimes.
I was merely pointing out that it's not correct to call something interplanetary if it's not. It's mostly false advertising.
>Are those the ones that go whoosh?
Does sarcasm prove your point?
Fair points :) We'll be addressing this in the coming months with increased work on the network stack (libp2p).
Just for one? Because, if you have the ability to do that, your girlfriend's phone has the ability to detect whether anybody nearby is accessing any arbitrary file or page. It just has to host a copy of that page and see whether anyone pulls it.
Unless I misunderstand your point, but honestly it seems like people here are engaging more in "gotcha" nay-saying than honest efforts of criticism... it would've taken you two minutes of googling to find out this is a non-issue.
If so you can just explain it (or report it).
You never host anything unless you want to; you don't host random stuff.
It's 2017, if you're using Google Docs or an instant messenger program and lose access to the backbone, you can't communicate with somebody who's in the same room with you. That's kinda silly. IPFS solves that issue.
IPFS is censorship resistance because it's a distributed protocol that can use a variety of transports. If I run example.com, people can DDOS it; it's much harder to do that when hundreds or thousands of nodes have the same content and you can connect to any of them. Sure worked in Turkey: http://observer.com/2017/05/turkey-wikipedia-ipfs/.
Filecoin is a cryptocurrency that will be mined by providing storage via IPFS.
Folks may want to read the white paper before they make assumptions about what's possible and what's hype: https://filecoin.io/filecoin.pdf
As long as IPFS requires replication to be voluntary on the side of the nodes, the argument of the parent holds.
They certainly have the funding to create a storage market that can rival what passes for distributed storage: https://www.coindesk.com/257-million-filecoin-breaks-time-re...
There's nothing preventing anyone from going through these same measures with IPFS or dat. It's just that you don't have to in order to get started hosting something.
Then explain that to the author of TFA, because they seem to imagine that websites and whatnot are just somehow going to be out there forever with IPFS, without having to rely on one's server. In reality, unless the content is popular and anyone's bothering to replicate it, it's going to still fall off the internet the moment the server's down in an IPFS-based web.
There are over 5 billion files hosted on IPFS and over 500 GB per day going through the IPFS gateway. Not bad for something that supposedly doesn't work and that's only been around for a few years.
When people can get paid to make content available on IPFS… well, that's going to to be a quite a thing.
"But who's going to pay for servers to retain random peoples' and companies' web detritus?"
The same people who do this now. Because nothing in this scheme says you can't have servers just like you do now.
The difference is that you don't need a server to get started collaborating. If you want to host something to have it available offline, that's a built in feature. If you and some friends want to host an event invite, grass roots, you can do that. No need for facebook, no need for big cloud platforms. Share around a web page. People will host it while it's needed.
Personally I'm more into dat than IPFS... but they each of their own use cases. I like stuff that's not going to be "permanent". There is plenty of room for stuff like that. Not everything has to be in some permanent public record for all time. We need more accessible ways to share stuff like that. We need good ways to share stuff privately. Skip this ridiculous idea that Mark Zuckerberg should be privy to everybody's private, personal information. No thanks. Share that stuff on LANs, on encrypted p2p connections. Keep it nearby.
If you want to publish, that's what IPFS is for. And if you want it to stick around, invest the resources to make sure there are servers, whether they be on digital ocean or a bunch of raspberry pi's plugged into you and your friends' walls, that are seeding it. That's on you as someone who's committed to publishing information.
Pretty much all the defenders of IPFS here give me the impression they haven't bothered to read the article hyping IPFS that's linked at the top of this page.
What are you finding to be inconsistent?
I'm thinking of a "server" as a dedicated computer connected to the network. With IPFS, if you want to ensure your data is available, seed it with one or more dedicated computers, like you would now. Seed it on digital ocean or amazon even, if you want. Nothing prevents you from doing this.
In the future, there will be new ways of incentivizing groups of people to seed data that isn't naturally viral. But nothing is stopping people from using the time tested, old fashioned ways in the meantime.
I don't buy the idea that IPFS stuff is inherently permanent, but I don't care. Even the way that IPFS handles broken links is way better than how they're handled now. With IPFS you at least get a hash of what you're looking for. That's a lot more useful than what you get now. The only thing you get now is "404".
I'm seeing a specific disconnect between the people who are into it and people who don't get it: they have ways of answering the question "why would someone want to host your content?"
The people who are into this idea realize that the content itself often carries its own incentive to share. Given the right infrastructure, a lot of stuff will host itself because people will want to share it. That's how bittorrent works.
Also, there will likely be stuff that falls out of fashion. The test of time will not disappear, but these technologies make it much easier for people who care about preservation.