Decentralized Web Primer
gitbook.com
gitbook.com
I don't think this is in the spirit of your typical Hacker News reader. HN readers are naturally curious, but with the volume of what comes across, it can be really hard to decipher if something would be of interest or value just based on the title or page 1.
For me, it's just not possible to read very deeply on everything that seems interesting, and it would be helpful to get a summary sentence or two for posts that are non-trivial.
"So you know how people use BitTorrent to distribute big-budget Hollywood movies with occasional mediocre storylines? What if you could distribute your own data on a similar P2P network, but view it directly in a web browser (among other HTTP clients)? You suddenly turn your seed box into a "web server". Plus, you can stream videos on it too. So your home computer is not just a "web server", it can act as your own "Netflix" server to publish your home movies. And you could do it in a secure fashion. You would need a few protocols (eg IPFS), concepts (eg Merkle Trees), and some freely available software to make it practical, easy, and secure. These tutorials will show you how to get all of those things working together to publish & stream to the world."
Watch a demo of this @ 7:00 in this video: https://www.youtube.com/watch?v=8CMxDNuuAiQ
He does a great job explaining e.g. Merkle Trees in a practical manner that made me immediately go "Wait, I could use this for X and Y?!"
Recommended watch!
Perhaps "An IPFS Primer" would be a better title?
this whole winner takes all mentality is taking the world in a really scary direction
What we need more than anything is to actually realize the beautiful dream TBL had for the WWW. HN is one great example but there should be a million more.
There are other factors of course: firewalls, asymmetric bandwidth, security issues, technical ignorance… But the client-server model does play a significant role.
If you want a truly decentralised web, you need to modify a couple things: get rid of HTTP, and use a P2P protocol instead. Generalise IPv6 and get rid of NAT. Give people symmetric bandwidth. Make secure software that is simple to implement, simple to use, and hard to misuse (it's not that hard dammit, just look at qmail).
Actually, just like "the cloud" is just someone else's computer, "P2P" is just someone else's server.
My point was that in a P2P architecture you must be a server, everyone else must be a server. And being a server is costly. There must be incentives for someone to be a server, and that's not easy. I actually don't remember a single P2P protocol or framework that has solved that problem yet. Perhaps Bitcoin -- for miners only, not for normal nodes.
The real problem there is the asymmetry of the bandwidth. If we had as much upload as we had download, we wouldn't have any reason not to shoot for a 1:1 ratio.
I'm not sure what you're getting at. The problem is only magnified with the web: there's only one seed. If a site is down, it's down, and you can only hope Google cache or the web archive got a copy.
That problem is never worse with P2P protocols.
I am also assuming that if someone cares at all about seeding something, they will make sure it's not down too often.
People can post videos to their own hosted pages already. We "need" YouTube because 1) people generally don't want to host anything, and 2) it provides connectedness (search, related videos, subscribers, etc).
P2P makes the first issue worse. The second issue is a problem of data federation, not client/server systems.
No, because it is expensive to host things, as mentioned in the comment you are replying to.
No they cannot. Here's an example: what if make a nice tutorial, put it on my web site and submit it here and r/programming? If the thing is well done, people will download the video, and the sheer amount of requests may be enough to render my site unresponsive, simply because I don't have the bandwidth.
So If I ever make a video, there's a good chance I host it to YouTube. Despite my reluctance to feed Alphabet. Even though I already operate a server.
> 1) people generally don't want to host anything
People want to publish stuff —that much is obvious. They want to avoid hassles if possible. And they rarely think about what centralised hosting entails.
That doesn't amount to "don't want to host anything". Granted, hosting stuff yourself is a major hassle these days. It doesn't have to be, though.
In order of increasing independence you could choose to: - Host it on youtube, with the benefits and disadvantages you already mentioned. - Host it on a public S3 bucket and link to it from a HTML5 video player on your web page. You pay for only the storage and bandwidth used. - Keep the video on your own server, but pay a CDN to cache it for you. Actual bandwidth use of you server is now very low again. - Host it on a (big) server in a colocation center. You would typically buy bandwidth in advance though, so you want to be able to roughly predict how much you need. - Contract with a fiber laying company to lay a dedicated fiber line from the nearest backbone to your home. (Alternatively, a large microwave connection) Buy several 100Gbit NICS to go with it and negotiate peering agreements with the major networks.
Most people don't want the hassle and cost of the latter options, but they are very possible and very legal. In your example with the programming video, if a ton of people want to see it then somewhere along the chain a lot of bandwidth is going to be used and it is _not_ free. All you do is shift who pays for it.
Exactly, I'd like my viewers to pay for it. There are limits to my generosity, after all.
> No they cannot. Here's an example: what if make a nice tutorial, put it on my web site and submit it here and r/programming? If the thing is well done, people will download the video, and the sheer amount of requests may be enough to render my site unresponsive, simply because I don't have the bandwidth.
He was already stating he didn't think hosting videos on their own page was the best solution for most audiences, in fact, nearly every audience today. Why are you arguing anyway?
I made a previous comment about this being a hard problem[1]. If competent ultra techno geeks can get hacked, the problem looks to be unsolvable for the general homeowner to safely run a p2p server.
Many recurring security+usability problems that nobody has solved in the general case. Examples... Heartbleed SSL bug that's undetected for 2 years before being fixed, social engineering, homeowner misconfiguration, p2p server data not being backed up, software updates as an attack vector for decreasing security instead of hardening security, etc.
If my grandmother asked me, "A friend said I can share cooking recipes if I install a p2p food wiki server, what do you think?" I would immediately say "No! Don't install it. I will find you a _centralized_ web recipes forum for you to log in to!"
I ran qmail on my home server in the 1990s to personally control my SMTP needs which should theoretically make me the biggest cheerleader for a "p2p food wiki server". Instead, it informs my position to stop my family members from installing p2p software. Think about why I would do that.
The "symmetric bandwidth" isn't the only underlying problem.
>That problem is never worse with P2P protocols.
Everything has tradeoffs -- including p2p. Otherwise, you have an incomplete picture that doesn't consider pros and cons. P2p is worse for latency, worse for analyzing a unified network landscape to identify and filter out hostile actors, worse for non-techie usability, worse for instantaneous propagation of software updates to fix zero-day exploits, worse for homeowner costs, etc. There isn't a "set & forget" p2p appliance you can build that solves all of that. BitTorrent gets around the negatives of p2p because the value of the data users transfer (pirated Star Wars movies or Adobe software etc) overrides the hassles.
That comment doesn't address the same problem. I'm fully aware that current software makes it virtually impossible to conveniently and securely host stuff ourselves right now. I am also fully aware that a general purpose computer can never be idiot proof.
What I was saying is, we should write special purpose software dedicated to a few chosen forms of hosting (mail, web, some P2P stuff for big files…). The software can be simple, secure, and easy to use. It just doesn't exist for the most part.
Also, one problem at a time. Wikis are great, but we're talking about allowing untrusted write access to the system, which is possibly the hardest problem of all. Let's start by single author publishing and build from there.
Fact is unless you're a Tier 1 ISP yourself, you're going to be beholden to an ISP to host your content. Your domain name will be beholden to a registrar. Your TLS certificate will be beholden to a certificate authority. You're going to be subject to every one of these organizations' terms of service, and history has shown with the examples above that they will exercise that authority to pull the plug on you if it suits them (especially if they're facing a mob of public opinion or the state puts a gun to their head).
DNS is fundamentally centralized.
I believe the context of "The Decentralized Web" here is meant to contrast the increasing centralization of content from FB, Google, Medium, login walls pay-walls etc.
Using old well-defined terms to describe new concepts just leads to confusion.
It would be like writing a story called “the history of the industrial revolution” that’s really about 3d printers.
You might want to look up the term "walled garden":
https://www.technologyreview.com/s/512316/facebook-and-googl...
Certainly, they’re in the infrastructure + surveillance business, and (especially for Google) that’s concerning.
Also, “Decentralized” is a technical term. Google runs a large, decentralized infrastructure, based on web technologies.
“Consolidation” is usually used as a business term, and is orthogonal to centralizing operations. For instance, many companies have outsourced payroll to one centralized company. That does not mean their industries are consolidating.
When people intentionally use words wrong, it is bound to lead to confusion.
No “Decentralized” is not a technical term. "Distributed" is a technical term, Google runs a large, "distributed" architecture.
Sure your laser printer can serve a webpage. But nobody cares! They don't care to the extent that your printer-page will most likely not even be accessible, given the default settings on your or your ISP's routers.
In practice, most of your - and everyone else's - web traffic goes through the same few services. It doesn't matter that each of those services is made from a thousand servers each - logically, they form a single unit, and the web has mostly a star topology. This leads to ridiculously stupid amounts of waste at the endpoints - if you and me are sitting in the same room and both want to watch the same funny cat video, we both download gigabytes off youtube, even though after you downloaded the video I should be perfectly able to stream it off your machine via LAN. We use CDNs these days, which are literally top-down dictatorship style attempt at forcing some minimum distribution into this centralized system.
The web is centralized. Your comparison with writing a story would be more accurate like this: it's like someone writing a story about "modern slavery" that's really about wage slavery. Yes, the social phenomenon of actual slavery has pretty much disappeared, but we've replaced it with something similar with similar issues, only one level of abstraction higher.
As for IPFS, I’d love to see them succeed, but as long as you have to know about merkle trees, hashing, etc for it, it’s not going to be any more mainstream than things like PGP or IRC.
Finally, they claim IPFS is good for archival. To me, that means it will be as good as putting a few hundred paper books in a few hundred different libraries around the world. With that in mind, how does censorship work with IPFS? If the government wants to rewrite history by deleting my archive, how do the mirrors avoid being tracked down, and told to selectively delete the undesirable blobs?
You have to know about this stuff to use ipfs like you have to know about low level packet routing to use the internet.
In social media: [Mastodon][0] is decentralized. [Patchwork][1] is distributed.
[0]: https://joinmastodon.org/ [1]: https://www.scuttlebutt.nz/
Could ISPs throttle or block IPFS content, for instance? The developers should start assuming some extremely aggressive environments, like China, and then go from there, because honestly, we don't know if 20 years from now a lot more governments would see China's censorship as a role model to follow. We're already seeing multiple countries in Europe considering it to stop "extremist content", the IPSs in Canada are now banding together to block pirated content, and the US is probably not going to be too far behind after the repeal of the net neutrality rules. And those are just the "democratic" countries. It's obviously much worse for Middle Eastern or African countries already.
So I just hope the IPFS developers will always try to develop the platform from an extreme resilience point of view.
"Swarm has a very strong anti-censorship stance. It incentivizes content agnostic collective storage (block propagation/distribution scheme). Implements plausible deniability with implausible accountability through a combination of obfuscation and double masking (not currently done). IPFS believes that wider adoption warrants compromising on censorship by providing tools for blacklisting, source-filtering though using these is entirely voluntary."
https://github.com/ethersphere/go-ethereum/wiki/IPFS-&-SWARM
I recall an IPFS dev saying this document was a pretty fair comparison, but it's been a while and I don't have a link.
>Could ISPs throttle or block IPFS content, for instance?
They wouldn't know what was IPFS and what was regular encrypted TCP, the only way I can tell IPFS traffic apart in wireshark is the port number. Once ISPs start blockings random encrypted connections people will get mad when their games or other applications stop working.
If we are talking about bad actors in the IPFS network, I reccommend you read their whitepaper. They layout a clever karma based system for incentivizing nodes to help others and punishing nodes that misbehave.
The extreme conclusion of state-sponsored censorship is not a black list or DPI. It is a white list. It is easy to restrict traffic to a small set of addresses that are rigorously monitored for any ability to proxy non-conforming traffic. Everything else would get dropped on the floor.
IPFS and the like sit too high in the OSI layers to combat state sponsored censorship in the long-run.
It makes me wonder what the target market is. If this ever got big, it would just trigger the next step in the arms race it is incapable of sustaining
But as long as we have ISPs and a common communications architecture, if we start using a content-addressable system, doesn't that help censorship? As a censor, I jump from having to censor all domains that may serve one particular document (which is difficult, as we can see with pirate bay), to just having to force ISPs to block urls with the hash of the document that I want to censor.
So we go from having to jump among domains, to having to jump among content hashes, which seems much less practical, isn't it?
In my mind, until we have some sort of mesh network with efficient cache systems, the decentralization topic seems (to me) that is providing answers to the wrong questions.
Maybe they changed the behavior in the meantime though, but IPFS didn't permanently replicate uploaded content in the past without some deliberate user action.
They have a research page on this: https://github.com/ipfs/research-CRDT
One of them is us (https://github.com/amark/gun), David reached out the other week, now we're trying to find time to discuss with Juan how to best integrate.
This "primer do the decentralized web" title is just bullshit.
Is it built as a one page anywhere already? Seems weird not to include this by default.
Highly recommend.
https://youtu.be/qZDJ1z0apVk?t=7786
Disclaimer: I helped organize the event where this talk is from.
Granted, nowadays you have things like the internet archive, but I don't see normal people going along with IPFS for dynamic content like social media as long as you cannot delete things.
[0]: Yes, I'm aware anyone can host a copy of my deleted page. But it will be served on a different URL, and it takes extra work to do, unlike IPFS or similar.