Achieving Scale in the Decentralized Web | Hacker News Reader

Achieving Scale in the Decentralized Web

pfrazee.github.io

66 pointspfraze16 comments

So all that's missing with the decentralized web is a centralized service to aggregate the decentralized streams?

I don't see what is being addressed here other than Twitter didn't make the client be your web server (push vs pull).

Sure the decentralized approach liberates the data. But without critical mass what good is that data?

Maybe the example is just not compelling enough to illustrate the real point here?

The decentralized approach removes a hard binding between the app provider and the user. When you post to Twitter, you add content to their site. When you post using the P2P web, you add content to a site you own. You retain ownership of the content; the content URL remains independent of any service; and the content remains local. You can move a P2P site to a new frontend application, and you can switch to a different aggregator for a global view. For that matter, you can skip the aggregator service, and use the pure P2P arch, and even crawl locally.

It's quite a bit more than "mak[ing] the client be your web server." It's about independent publishing.

Sure I get that. What you call "The P2P Web", I call "The Web". Now if you want to talk about P2P Twitter, that is a different animal.

Thanks for the article and the project. I also was skeptical at first of requiring one or more centralized web crawlers to add functionality based on global knowledge of posts. Still your proposal has the advantage of user control of content. To some degree you can get this with the current web by "following" the Feeds of certain blogs and construction a timeline for yourself.

What do you think of GNU Social? I am a fan and enjoy it.

Off topic but, I would like to thank the people who created the Decentralized Web Conference in SF in June. Really nice conference, and I look forward to the next one.

> To some degree you can get this with the current web by "following" the Feeds of certain blogs and construction a timeline for yourself.

Yes definitely, the problem with the current Web is that you cant deploy sites cheaply enough. P2P tech makes domain allocation free and fast, so it can fit into a typical application's flow. (The Dat protocol also caches data really well, so read latency is less prohibitive.)

> What do you think of GNU Social? I am a fan and enjoy it.

What I know about GNU Social, I like. I think its impact will be limited because it's still the old model of federation that binds user identity & data to a specific server. But I still support it.

> Off topic but, I would like to thank the people who created the Decentralized Web Conference in SF in June

See you at the next one!

While the decorrelation of user data and application provider is important, I feel like the proposition is relying a bit too much on a centralized service. Perhaps reusing some ideas of OStatus would be helpful ? Something like that:

- UserA starts a feed at dat://UserA, and publishes stuff in Activity Streams format

- UserB follows dat://UserA

- UserB comments on UserA feed: it creates dat://CommentsFromUserBonUserA, and publishes a comment in Activity Streams format there

- UserB pings UserA telling them about dat://CommentsFromUserBonUserA: UserA is naturally interested, so they just subscribe to this feed, and republishes stuff into dat://UserA

- UserC subscribes to dat://UserA. They get UserA's posts, along with UserB's comments, without every subscribing to dat://UserB. All activity about UserA is in one place, completely under UserA control, with no dependency on a centralized service

- Same can be done for mentions of UserA

This obviously assumes pinging is easy, which it isn't. Perhaps some polling of a DHT ? It also doesn't solve the issue of hashtags... but it makes dependency on centralized service less important

A purely p2p ping would help with non-centralized discovery, but it isn't easy. I'd be receptive to a proposal.

Web Crawling is a reliable federated system that doesn't impede user freedoms. The crawlers are so fungible that anything centralized about them doesn't really concern me.

> There’s no walled garden for Web crawlers. Their datasets can be freely replicated from the public Web.

The same can largely be said of the normal web when it comes to Google, but that doesn't alter the fact that they have a monopoly. Brand recognition, economies of scale and similar underpin their position. Can a decentralized web prevent monopoly?

>The same can largely be said of the normal web when it comes to Google, but that doesn't alter the fact that they have a monopoly. Brand recognition, economies of scale and similar underpin their position.

Well that's like saying that GNU will never succeed because Windows and Mac have such huge market shares.

Obviously not true.

I guess we could ask Google to stop outperforming all their competitors.

Two thoughts on this:

- Maybe someday we'll find a way to do fully distributed services reliably. None of the global blockchain solutions are good enough, by my standards. What we can do is remove the friction to creating competition, which the P2P stack and other techniques (like the old Web Intents spec) are for.

- I would like to see home servers become a thing. Put ten terrabytes in the corner, run a bunch of crawlers on it, include some non-public services. You can't beat google on searching the web as it exists right now, but google cant search private data.

google bundles ranking with crawling. if I just want meta data about a set of sites that's different. Also, if your talking real-time web post-date(time) is pretty useful data for a feed.

"If we put all economics aside, it's obvious the web should be organised this and this way..."

What I'd really like to see is decentralised web proponents (of which I'm no longer one) put out some ideas for how their model is going to support economic activity of any interesting kind. Yes, selling advertising by centralising eyeballs is shit, but show me something better.

The alternative is to either wait for capitalism to finally be over (spoiler alert: they're planning another season, starting January 20), or to get a second gig as a gardener to fund my web habit.

We shouldn't let unresolved economics stop us, but here's what I've written previously: https://www.reddit.com/r/Rad_Decentralization/comments/5j878...

> Subscription services will still be an option, even in a P2P Web, but we are trying to reduce the natural monopoly of the dataset (no walled gardens) so that might make affect service revenue by increasing competition. I'm not sure how destabilizing it will be.

If a service can be destabilized by the lack of a walled garden to lock in users, then it's in society's best interest for that service to be destabilized, because that means there are other competing services that users want to use, but can't because of the size of the existing service's network effects.

The status quo of services competing mainly by the sizes of their network effects is completely hostile to innovation, experimentation, and user choice. Any efforts that could disrupt this status quo has my full support!

Completely agree

NNTP? Wasn't that kind of a solution for this?

Reply on news.ycombinator.com