The Vision for Decentralized Web | Hacker News Reader

The Vision for Decentralized Web

tautvilas.lt

106 pointsbrisky51 comments

This a problem I’ve something I’ve been working on my free time. The biggest problem with the decentralized web right now is that setting up a server is a huge pain and your average person lacks the knowledge and resources to do so.

I’ve been working on a simple music streaming server called mStream. The goal is to make a server even my mom could setup.

I publish binaries for every major OS which means you don’t need to rent a server or buy more hardware. You can just run it on your home computer. mStream is configurable entirely through the GUI, so you do not need to open a terminal to get it running.

The hard part is making that server easily accessible on the internet. Setting up SSL and dealing with NATs still requires tech knowledge. I’ve been making some beer money selling hole punching software, but that still requires me to host a server so other people can access their servers.

Your first paragraph is absolutely correct. The issue I see with your solution is that there's no fault tolerance, so the service is likely to be unreliable, and then of course there's dynamic IP which means it will be difficult to connect to from the public internet.

Honestly, the real issue is getting people to pay for things. Running web services has a cost, but it's hard to "see" it. If you're mom is going to want that server to be available 24/7, then she has to pay for a computer to run it on and an internet connection (which she has probably already done) as well as the electricity to keep it all powered on 24/7, and that doesn't add any fault tolerance to the system. If you can convince her that all that cost it worth it to stream music, then why can't you convince her spend a similar amount to let someone else deal with the hassle and get better reliability?

The best solution is to simply pay a service to do the hosting and take advantage of economies of scale to provide an always-on service at a great value.

"free" services that aren't actually free are the problem.

In fact I would be interested to see if it's possible to protect consumers by regulating marketing of software and services as "free", where it can only done if strict data handling policies are applied.

I saw someone mention this a while ago and at first I thought it was stupid but over time I've come to agree:

Asymetric internet connections should be forbidden. They assume that users consume more than they create, and the internet should be a place where everyone has equal footing. In addition, IPv6 firewalls in routers should either be disabled by default or have a forced API to selectively disable them [^1]. (IPv4 for the "decentralized internet" is pretty much a lost cause because of NAT). In my opinion, this should be a part of net neutrality. The current definition of what constitutes "internet" is far too loose.

Many applications of decentralized hosting of websites or other content is much harder because most people have a very shitty upstream. If you could use all smartphones around the world that are currently connected to WiFi and an outlet, you could host whole websites (or at least chunked static content) over those. Currently, this is unrealistic because you can't NAT punch for every 100kb chunk and you won't get any good speeds.

[^1] My router by defaults blocks all incoming IPv6 connections to devices

Tor is often a valid workaround for both NAT and SSL.

>dealing with NATs still requires tech knowledge

Have you considered MiniUPnP library? It's very easy to use. Of course, the modem has to support UPnP(the most of are) and has it enabled by default(not sure how many are, mine is).

I have, and it works well for some people but not everyone. You also get a problem where these ports close after several hours, leaving the user confused to why things just stopped working.

I'll be interested to see where these distributed trends go. I too have been working on something to solve a problem I face; "information" sharing and storage. Fundamentally I want a public wikipedia focused around small pieces of information (though I'm still toying/dogfooding with the UX), but more importantly I don't think it should be a centralized entity.

Oddly though there's something I don't like about building on top of a blockchain. It feels strangely walled off. A distributed system with all the problems of a closed off startup. I'm likely being unfair, but I'm just saying how I perceive them. Despite being interested in this area, including working on it personally, I just have no interest in joining one of these blockchain things. I don't honestly understand what I or my hypothetical users are getting out of it.

I really like the Scuttlebutt protocol in concept; it feels very.. "email-like". Something that is a mix of federation and self-hosting. Scuttle or maybe ActivityPub seem to have a good feel for decentralizing. Blockchain (as I've seen so far) however.. eh. Am I being unfair / short sighted?

Some decentralized discovery method for interesting tech content would be nice. Something simple, like if you could put a list of websites you like to:

    https://my.domain/.well-known/distributed/recommended.list
    -------
    https://friend1.domain/ Blogs about electronics
    https://friend2.domain/ Blogs about knitting
    -------

    https://friend1.domain/.well-known/distributed/recommended.list
    -------
    https://my.domain/ Blogs about hacking SBCs
    https://friend3.domain/ Nice website
    -------

And you could write a bot that would walk through this, avoid loops, and construct a graph, and give you a search or RSS. RSS feeds could be also discoverable.

    https://my.domain/.well-known/distributed/my-feeds.txt
    --------
    https://my.domain/.well-known/distributed/feed-blog.xml My blog posts
    https://my.domain/.well-known/distributed/feed-announce.xml My announcements
    --------

Just something that helps with discovery of interesting content in a decentralized way, outside of search engines, aggregators, HTML pages, etc. You would be able to select search distnace from certain domains, blacklist domains, etc.

And if you assume person == domain, this would build an open, distributed, searchable social graph (of people with domains :) and something to say). You'd be able to see weights of domains by how many people recommend them, etc.

There would be no central storage for the graph.

That is something I have been thinking about too. I imagine this could work as a distributed decentralized db where you publish some link and sign it with your domain key. A client could easily select the most submitted urls from DB and also 'follow' certain domains(users) that submit the most interesting urls. I would love to discuss and validate this idea more.

Yeah, one problem is that on the current internet, if people are not constantly posting low value crap on social media, they are hard to discover, and they're ranked down just for inactivity. But this system would support ranking just on the number of domain recommendations, and people would be able to make tools to do their own ranking (eg. how many people I recommend recommend domain X would be my rank for domain X,...), etc.

But I think, it would need to be decentralized, to avoid single DB somewhere that could be abused and manipulated, or lost.

But if you have to use a tool to read the graph from the web yourself, and people have to upload recommendations to their own server, and have them directly associated with their domain, that may encourage saner behavior, too.

If someone wants to make a centralized public search for this graph, so that people who don't see value in building their own index/ranking could search for content too, that would be possible.

But the key aspect would need to be the distributed nature. It would not need any cryptography either, just a bunch of text files at the simplest level.

The graph would be out there on web servers, the creativity would be in creating the tools to scrape it, rank stuff, and searching through it. And that could even be done in some incremental way (without the need to scrape it all at once).

Totally agree. Ping me on connect@tautvilas.lt if you would like to discuss this idea more and try to develop it :)

This is a really cool idea. I might spend some time hacking on something like this after dweb camp. If you're on Scuttlebutt I can CC you on my progress.

You should check out blockstack. The platform does not use blockchain to store data, but only to create a more robust domain name and identity solution. The data is stored on encrypted server storage. I think this is a nice combination of decentralization and blockchain at work.

"Blockstack PBC, a public benefit corporation at the heart of the Blockstack ecosystem, has raised $50M+ in capital to develop the core protocols and developer tools for the platform."

What is decentralized about a venture-funded vanity project?

Muneeb from Blockstack here.

There was a HN Launch discussion recently that might be relevant here: https://news.ycombinator.com/item?id=20413420

The decentralization comes from (a) open-source GPLv3 code, (b) permissionless blockchain, (c) enabling users to host private data lockers, among other things.

The discussion on decentralization in our recent SEC offering circular -- page 49 and page 69 at https://stackstoken.com/circular -- is also relevant here.

From page 69 of your circular:

"At some point when this decentralization process is complete and there is a healthy ecosystem of applications and users on the network, Blockstack PBC expects to develop new business models"

This is the Field of Dreams approach to reinventing the internet economy.

Also from that circular: "core nodes are hosted on Microsoft Azure, Google Cloud, and Amazon Web Services, or AWS, which provides us with computing and storage capacity. Some elements of this complex system are operated by third parties that Blockstack does not control and that could require significant time to replace. We expect this dependence on third parties to continue"

I'm struggling to understand what is decentralized about any of this. From my vantage point, you got VC to pay you to run a project on the cloud by saying "blockchain" a lot.

Blockstack engineer here.

The protocols under the hood ensure that each user chooses where their data gets stored -- all that's required is that wherever it is, it's available via HTTP(s) so it can be loaded into your Web browser. As such, Blockstack-powered applications hold no user state -- the user takes their data with them, and "brings their own storage" to the application. The blockchain component acts as a discovery mechanism, whereby any user can discovery any other user's public keys and storage locations, given a username.

Blockstack PBC gives users a small amount of free storage space to help onboard them, and runs some publicly-routable Blockstack nodes. However, nothing stops you from electing to store your data in your own storage provider (e.g. S3, Dropbox, your personal server, etc.).

If I could upvote you more than once ...

Have a look at Blockstack's gaia protocol for storage, it is free and open source. And you can also use it with other decentralized identity providers. If you don't like the company, you still might like their vision.

the problem with decentralization is that, for an average Joe, it's just a bunch of loonies wearing tin foil hats who have something to hide, and besides, what has Facebook done to me (average Joe) personally? People, ultimately, don't care about privacy that much, not until they see obvious consequences of corps having their data that affect them greatly in their lives. Privacy against employers/old school friends/fiancees is something they care about, that's why Snapchat, Instagram Stories and private groups are replacing the public newsfeed. Decentralization must offer some clear advantages over fb, that an average person might appreciate. Those advantages must offset the UX problems, of which there are plenty, most of them pretty hard to fix. In my opinion, the worst one is the ubiquitous use of smartphones over PCs, as any decentralized stuff on smartphones is a nightmare, mostly due to systems relying on servers for stuff like push and background app refresh. This is particularly important on iOs. Also true decentralization, NOT federation, with checks and balances, possibly with nodes that only connect to other nodes that have signed a sort of license / legal agreement, is the only way to ensure it won't turn into mail, AKA one big Gmail that doesn't care, and the rest which doesn't matter.

Good point. I think that's the fundamental problem why most people don't care about internet privacy or mass surveillance. It's invisible and very easy to pretend it isn't there. I'm not sure what can be done about that.

To be honest, I don't care that much personally. I know the theoretical risks, but the things I get for not caring are just too great.

> the problem with decentralization is that, for an average Joe, it's just a bunch of loonies wearing tin foil hats who have something to hide, and besides, what has Facebook done to me (average Joe) personally?

I agree, but I think there's a middle ground that fits everyone. Eg, I really like federation because I feel it's a quality middle ground _(depending on the implementation)_. It is what I am targeting my projects distributed implementation on.

Roughly, I am designing an application similar in concept to that of the ActivityPub federated protocol (and may use ActivityPub in the future). I want to support first-class self executed tooling, like git locally, while also supporting a federated multi-tenant solution to support the people who don't care.

Currently my thought is that people should be able to pick up their data and walk somewhere else with it if they like. Self hosting with federated multi-tenant, seems like a natural solution to the problem. My implementation _(goal, at least)_ doesn't "invent" anything, uses standard web tech, and supports both federated and solo usage. I'm being purposefully vague for anonymity sake on this account, fwiw - apologies for that heh, but hopefully it doesn't matter to this discussion.

> In my opinion, the worst one is the ubiquitous use of smartphones over PCs, as any decentralized stuff on smartphones is a nightmare, mostly due to systems relying on servers for stuff like push and background app refresh.

Due to the nature of what I am creating mobile matters as well. You're definitely right in that mobile is a pain here. My goal is to "first class" support mobile too, and my "protocol" would hypothetically support first class mobile, but given that storage and cpu are quite limited a client->server architecture will definitely be useful. I plan/hope to make mobile work in a hybrid manner, supporting both on-device and a federated server, as on-device will be important for offline functionality.

> Decentralization must offer some clear advantages over fb, that an average person might appreciate. Those advantages must offset the UX problems, of which there are plenty, most of them pretty hard to fix.

I agree, which is why I don't think decentralization itself will be a winning feature. Personally I don't think there's anything wrong with what Facebook is doing - what's wrong is that we can't take our data off FB and move to a platform we're more aligned with. One maybe that we pay for, or one I self host, etc.

What guarantee do you have that your thing won't turn into the "Gmail and the rest of them" problem? Basically you need money and tech skills to run a server. Small, amateur servers might have problems like much more frequent downtime, being paid or small and hard to find etc. In that situation, some big corp, perhaps Google or Facebook, could create their own server, free for everyone, but with no privacy and lots of tracking. People will migrate, and then, slowly but surely, the trap will start springing. The service will be locked down, federation will be disabled, most data export functions too, same for the API etc. The quality will slowly degrade, but this willbe the point of no return. I see that happening with Gmail already.Mail was great, but we need more. Decentrealized protocols should be based on constitutions of democratic countries, as they've learned their lesson already. It's not enough to make a country democratic now, we need to secure it against future hostile attemts at takeover. Same needs to be done for decentralized/federated protocols.

> Basically you need money and tech skills to run a server.

Yes, you do, and also an ISP that will allow you to do so. I'm lucky in that my ISP allows me to run almost anything.

However as regards a server, you don't necessarily need a big, expensive machine. I run a low-traffic web server on a Raspberry Pi 3 and it recently had a mass visit from HN readers when I did a "Show HN" - no problems. With the Pi 4 there are lots of possibilities with the extra power etc. Why not start small and see how long it takes to outgrow your machine - it may take longer than you think.

> What guarantee do you have that your thing won't turn into the "Gmail and the rest of them" problem? Basically you need money and tech skills to run a server. Small, amateur servers might have problems like much more frequent downtime, being paid or small and hard to find etc. In that situation, some big corp, perhaps Google or Facebook, could create their own server, free for everyone, but with no privacy and lots of tracking.

This is not a problem I'm trying to solve, as I have not heard/thought-of a solution I like. I also feel it is a minimal problem.

Facebook is a problem in my view. Gmail, not so much. Why? Because I don't use Gmail, and I have minimal to zero issues caused by the fact that I don't use gmail. Federation seems to work well here.

> In that situation, some big corp, perhaps Google or Facebook, could create their own server, free for everyone, but with no privacy and lots of tracking. People will migrate, and then, slowly but surely, the trap will start springing.

I'm not sure I've seen that with something like email. Though, to be honest I'm not sure I've seen anything like email, either. It sort of happens with email in the sense that without a large email provider spam is difficult to beat. Furthermore large providers are distrusting of non-providers (ie, running your own email server), but this to me seems less of a problem with federation, and instead a problem with email itself - one I'm not sure how to solve. Spam is a difficult beast.

> I see that happening with Gmail already.Mail was great, but we need more. Decentrealized protocols should be based on constitutions of democratic countries, as they've learned their lesson already. It's not enough to make a country democratic now, we need to secure it against future hostile attemts at takeover. Same needs to be done for decentralized/federated protocols.

I don't understand what you mean tbh - but if your democratic countries create standards like email, federated and distributed, then I don't have any complaints. Though, to me (as a hypothetical dev) it makes little difference that you added politics on top of it - I just care that it's federated and I can own my data/etc.

Chicken and egg: Why would anyone leave Facebook/Twitter/CorporateTool for this or any other competitor?

Also, what's to prevent this from being boiled down to another set of Facebook-like/Twitter-like/CorporateTools, just as the early web was contorted into today's interactive TV?

I would like to know how many people still use the web in comparison to how many people use the Internet. My guess would be that normal people are moving to using specific mobile apps for everything they need.

Some of the Blockstack apps work on mobile as well (I haven't tested them all but a saw some good progressive web app designs).

Personally I think that the most practical decentralization solutions are going to be p2p.

And also I think the big challenge we have now is for people to really work together and support some of the ideas out there. There are actually a very large number of projects in this area. Most of them are both fairly useful and also fairly unpopular.

I think the key is going to be to get some social momentum behind some of the better solutions.

Out of topics I'm in no way am a tech-savvy or Internet security/ piracy etc. advocates by any means (although I've been looking into it). Q the article is straightforward to follow and comprehend , that I have not once google up any technical words. I like the writing in this.

A blockchain is just trading one walled garden for another.

The true vision of the decentralized web is reduced friction by having no gardens.

Sites like notabug.io (p2p Reddit) and d.tube (decentralized YouTube) are doing this, no need to login, no pay wall.

You don't need a blockchain to decentralize, just stuff like WebTorrent or GUN.

Gah!

While reading the article I followed links and secondary links, and searched, and I now have 18 resulting tabs open.

The last one is this: https://en.wikipedia.org/wiki/Special:Search?search=decentra...

and the first link on that search result page is to TBL's Solid project, which seems to be Yet Another DWeb effort.

Should I think of this, and other Dwebs, as an eventual global replace for FBook et al?

Or is/will it be more like superpowered personal web pages? And will my aunt have to run her own ... node?

Author here. I would like to address some of your questions:

1) The web is decentralized by design - we only need to 'upgrade it'

2) We need to create tools that empower people to leverage decentralized capabilities of the web (so that there would be no need for aunt to run the node)

3) I imagine we should not think about DWeb as some sort of FB replacement but more of an ecosystem where people feel that they have a choice and don't need to give away their content to facebook and other centralized platforms.

It is not clear how this has to be implemented but I feel that some more privacy oriented and data-ownership oriented platforms will emerge in the next five years.

> It is not clear how this has to be implemented but I feel that some more privacy oriented and data-ownership oriented platforms will emerge in the next five years.

Thanks. I look forward to opening another 18 tabs in five years. :)

AFAICS, neither Blockstack or dpage.io are "decentralized". Note the login page. Ask what happens if dpage.io site goes down. This is just a website. The fact that this site uses Blockstack's centralized servers doesn't make it "decentalized" any more than using AWS would.

Muneeb from Blockstack here.

Dpage can be easily converted to a Desktop app. Most of the app logic is client-side anyway and hosting at a domain is mostly for convenience.

Blockstack's "centralized servers" doesn't really fully capture the technology at play here. The pointers to user data lockers are stored in the Stacks blockchain, and users can host their own private encrypted data lockers where they want. Blockstack PBC does provide a free service to host data lockers, as a way to help bootstrap initial usage.

It's fine to have some default servers for discovery bootstrapping (as Bitcoin does) but any subset of Bitcoin nodes can go down without the network failing. Is this true of Blockstack? If every Blockstack controlled or financed server went down, would the network continue to work?

Of course you still need to have servers somewhere. The decentralized parts are these: blockstack identity (which is stored on blockchain) and the fact that all user data is not stored on one server, but on multiple nodes. The user can choose what node to store their data on. It can be your own hosted node too.

If dpage.io goes down all the authors still have their pages on their storage. It is just a question of spinning of a new page that aggregates the pages again. Authors own their page content.

They seem to have forgotten Step 0, create a viable business model that can pay people.

Many Blockstack apps are about managing user data. If you want to make profit from this data then you stay where we are at the moment with data capitalism...

You need money to build and run the apps. Source: I do this for a living.

The operational costs are considerably lower with Blockstack, since the app doesn't host any user state (and in many cases, no state at all). Most apps don't need a database of any kind -- all the business logic runs client-side in the user's web browser.

Event if operational costs going to be 0, it's not a viable business model to not have profit.

I never claimed otherwise ;)

However, I will claim that the lower operational expenditure (and lower liability for holding user data) makes profitability on-the-whole easier to achieve in Blockstack-powered apps.

needs more vision. decentralization alone won't get us out of this nightmare. even if we design the perfect replacement system, we'll never get the mainstream off the old one - not without nuking Moab Utah.

> In order to implement the vision of decentralization, first we need empower people to easily publish and own their content on the web.

Why? If someone wants to publish, they'll learn what needs to be learnt. If they are not willing to do so, they don't really want to publish, so why empower them? (This is a serious question.)

Because already have centralized services that easily do that. You can't attract users without this.

I think equating "the internet" and "the web" is sloppy; the author does this in the first paragraph and throughout the article.

> It [The Internet/The Web] took off with hackers running independent servers in their basements.

Is that true? Maybe if we're talking "the internet", things like MUD, IRC, Usenet, or BBS servers were run by individuals, but looking at the "List of websites founded before 1995"[1] it seems like there were big players outside of labs/universities pretty early on in the history of The Web. I'm asking a legitimate question here: was www successful (at least in part) due to websites from individuals (let's maybe say "students on university networks" rather than "hackers in basements") over large institutions (like the magazines)?

> The informational landscape today shifted from people sharing personal websites to sharing social posts.

Again, I question the accuracy. Did more people have personal websites than BBS/Usenet/Web-Forum accounts?

Regarding the dpage.io site this article is promoting:

> As you can see, there are no ads, no clutter, only information that was created by the user.

This is a problem I think the "distributed web" community has to deal with: What if the user wants to monetize their content, perhaps with ads? Either people are going to do some "pay per view" of articles on your network (which seems to be antithetical to "information for the masses" that I have a feeling most decentralized developers identify with), or they use ads. Want your ads to do better? well don't fund the network that has reader anonymity built in.

> The domain name system enables any person on earth to have his human-readable address on the web. This system is decentralized by design and is owned by nobody.

So they want to fork DNS from ICANN so your identity is always linked to your website?

> Dpage.io is not a website builder, and it is not a blogging platform.

Design your own website, upload it to us and we'll host it for free.

Reading the contents of the strange little iPad on https://dpage.io/ it says:

> Your user profile details along with your data are by default stored on a free Blockstack's Gaia storage hub. DPAGE doesn't store your data on its own servers. You can run your own storage hub on a server of your choice.

So it is that kind of distributed server: the one you run yourself.

<aside> The phrase "a free Blockstack's Gaia storage hub" is strange and feels very corporate, being sure to indicate that it's not just any Gaia server, but the one OWNED by Blockstack. Sometimes it's the little things that show you how much control the marketing department has. </aside>

I'm not going to comment on their blockchain stuff.

I'm only criticizing because I think people are putting lots of effort into solutions that are misguided: "The people WANT to have their own websites! This is the platform they'll want for sure!". I very well may be missing the point here, but inaccuracies and romanticizing the past lead me to skepticism.

XAMPP[2] is still around after 17 years, (last updated 4 days ago), which I remember being super easy to setup your own site, and I have a feeling it's even easier now. CMS platforms like Wordpress.com, SquareSpace, etc, do allow you to own your own content, easily use themes/plugins, and backup/export everything if you want to move (for a price, because they're providing a service, which is reasonable).

I think the most interesting decentralized platform is dat/beakerbrowser, because

A) There's already a tangible working product (Beaker)

B) There's no login/user-admin: your browser upon startup is a node in the network

C) Backwards compatible with http(s), so it's not like there's a hard fork of the DNS namespace and links no longer work.

D) No concept of privacy/encryption (that I'm aware of). You really don't own your data, you publish it and the system is designed to allow anyone/everyone with the link to copy it, change it, host it, seed it. If you don't like that, don't use it.

Of course you have to seed your site, so again there's a chicken and egg problem of hosting your own server (but if it's just running the browser until your friend says they've downloaded the site, it's easy, right?)

I didn't mean for this to turn into a rant, but I guess it did.

I wish dpage/decentus/blockstack/gaia good luck because I think distributed webs do have a future and every player can bring something to the table, but I hope their message/intentions become a little more focused.

[1] https://en.wikipedia.org/wiki/List_of_websites_founded_befor...

[2] https://www.apachefriends.org/index.html

Reply on news.ycombinator.com