Hacker News Reader

(Premise: I love decentralized systems.)

> A decentralized web would be vastly more efficient in terms of resources. A lot of the data could be deduplicated and a lot of backbone traffic could be avoided.

Decentralization usually brings more duplication of data. Think of email: copies of the same email are redundantly stored in many servers. And this is what we want. I have received the email, I decide if and when to delete it. Or Bitcoin. It is "decentralized", but if you want to run a normal client an be part of the federation you will need to spare quite a bit of disk space to store the exact same 90GB that comprise the blockchain.

> Right now, if a dozen people around a block watch the same viral video, all of them will fetch it through the backbone, which is obviously wasteful. If they want to store the video for later, they will have a dozen copies wasting space on their drives, which could again be reduced by more efficient algorithms (eg. erasure codes).

At the same time we do not want people to know that we are watching this video. We can get that video from our neighbors only if the ID of the video leaks somehow (if you reply "TOR", I will answer "exit nodes know"). But that is just a side problem.

The main problem is dynamic content. Almost all my open browser windows contain pages that are generated "just for me" (github, gitlab, travis, HN, a work-related 2TB ownclound installation, amazon, google search, ...). These are the pages I care about, and, sadly, these are the pages that are most difficult to decentralize in acceptable ways. Actually some of them are meant to be centralized (amazon or google search, for example).

I'd love to see more decentralized or semi-decentralized services (planets and their OPML files are great, for instance), but one has to admit that some problems just ask for centralization.

PS: weird, I had to write "google search" instead of "google". Using just the word "google" felt somehow wrong. Brand dilution, I suppose.

Tor actually only uses exit nodes if you want to connect to the regular web, if you fetch a video from a .onion address the encryption is end-to-end. And there's also Freenet.

Github/Lab and HN actually seem pretty easy to decentralize - git already is, and the rest are essentially feeds, much like email/newsgroups/RSS/FMS. It's mostly a matter of running the aggregation code in the client, rather than on the server. Google can also be decentralized and federated - see YaCy. Amazon, there wouldn't be much point, since the service would still be contracted with a single party.

It's not a trivial job (in the "I could do that in a weekend"), but it doesn't seem architecturally hard considering the existing distributed platforms.

> Decentralization usually brings more duplication of data.

While this is true this could be vastly reduced if the internet supported source-specific IP multicast (SSM). That way even a small box in someone's basement hosting a single copy of some niche content could serve thousands worldwide.

Alas, multicast is hard and incentives are not in favor of making it available to the end-user even in limited scopes.

In terms of computation and storage, maybe, but in terms of the network bandwidth, perhaps not.

The current physical topology of the Internet is already heavily biased towards backbone-to-edge connectivity, and is directionally asymmetric.

By comparison, edge-to-edge connectivity that doesn't go through any backbone is much more limited, and edge nodes often don't have the upload bandwidth to effectively deliver content.

Plus, CDNs and other distribution networks already effectively put easily cachable content closer to the edges to relieve backbone congestion.

Is it possible we already have an entrenched system in place that squashes the possibility of a decentralized internet before it can get off the ground?

Not only is no one economically motivated to share traffic towards the edge, no one will be building physical network capacity for it either.

I wonder what projected bandwidth numbers look like (if, for instance, youtube or netflix was completely distributed) - is the edge network close to being able to handling this?

I think edge to edge connectivity being limited is mostly a software (routing protocol architecture) issue.

For example, I live in an apartment and the ISP has a router installed in the building. Coax cables are running to each apartment. I can see 15+ wifi networks from my home. Yet to fetch the exact same front page of a news site my neighbor is reading and has a cached copy of, I am routed through the backbone probably half way around the world, or to a CDN at the other end of my country at best.

I don't think photons really care which way they travel but the shortest the path the faster they will get there and the less photons we send the cheaper it ends up being.

If the argument is that this saves on network hardware costs, then you'd think ISPs would have some incentive to invest in decentralization as well.

But I'm not so sure about the thesis that it's just a software issue. It seems wasteful to sling electrons through a backbone and back when those bits are (physically) close by, but the cost here is not the distance those bits traveled, but the marginal cost of the physical hardware to deliver those bits to your doorstep.

It's probably much cheaper, on a cost-per-bandwidth basis, to beef up the existing backbone, because significant economies of scale exist.

OTOH, there's a significant fixed cost every time you try to widen edge links (it's very expensive to dig up and replace residential cables).

My ISP would certainly love if my whole local area (building/street/district/whatever) only fetched each unique piece of data once through their network and then distributed it locally via mesh networks. It's just that they can't make it happen by themselves. It needs to be supported by network protocols, which need support from content providers and content consumers as well, so it goes way beyond the reach of ISPs.

> The current physical topology of the Internet is already heavily biased towards backbone-to-edge connectivity, and is directionally asymmetric.

Couldn't agree more. The internet as we have it today is set up for decentralized consumers and centralized producers.

There is one untapped benefit of the current system which is that there is low latency on the edges. I'm not sure what the killer app to make use of that would be though.

Not every thing is in the same backbone. Netflix has installed datacenters at ISPs as cache. Which is more efficient :)

Yeah, I can't find a map of Netflix's caches, but it looks like YouTube has just about every large city in the world covered except in China, Iran, and Turkey. (bottom map at http://peering.google.com/#/infrastructure)

> Whenever you buy a product from a company that advertises on facebook, google, whatever, you are effectively paying for it.

In my opinion, it's a bit more nuanced than that. When you purchase a product with cash, no personal data is exchanged. The transaction is final and complete. When you 'pay' for services by exchanging personal data and metadata about yourself, that transaction is persistent and the vendor continues collecting payment indefinitely.

With store loyalty cards and even in-store Wi-Fi tracking and other tactics, even cash payments can be viewed as less anonymous, because the vendors prefer to continue to reap the rewards of profiling and tracking customers. My point is that this isn't a black and white issue, but there are some important distinctions to make about payments in cash and payments in data, or a mixture of both.

The way I see it, when you use facebook (share personal data, view ads) you are not paying for anything in any way. You are just making counters go up (number of ad views, amount of data for targeting, etc) and thereby validating to companies that it's worth paying facebook for ads. Then, when I buy groceries for $100 and say $1 of that is used by the grocery store to buy facebook ads, I am paying for your facebook usage. And I'm not even a facebook user.

> Whenever you buy a product from a company that advertises on facebook, google, whatever, you are effectively paying for it.

Or in social media's case -- we are the product that they make money off of.