A decentralized web would give power back to the people online | Hacker News Reader

A decentralized web would give power back to the people online

techcrunch.com

710 pointsendswapper348 comments

So nobody can "give the power back" to anybody, but people can stop giving their power away to others. We all have power, and we manifest it through our actions and choices.

Some say knowledge is power, but in order to gain knowledge you have to make that choice. One can try to educate people about the benefits of hosting their own data, but unless they make the choice to listen and understand it's all for nothing.

I tend to agree with others that posted here, that in general, people are just not interested. It may be too hard for them to grasp the real implications of giving all those informations about them to third parties.. They may also consider that it's too hard to handle all those problems by themselves so they're willing to pay the prices...

I don't think that the change can happen unless the people who do understand, do something about it, but usually these people are more interested to cash in on the ignorance of those who don't...

>>I tend to agree with others that posted here, that in general, people are just not interested. It may be too hard for them to grasp the real implications of giving all those informations about them to third parties.. They may also consider that it's too hard to handle all those problems by themselves so they're willing to pay the prices...

It's true that many users who give away their data are not cognizant of the consequences, and there are others who understand but do it reluctantly because of peer pressure or usability. But there's a huge swath of people who fully understand the privacy implications of giving away all their data in exchange for the products they use daily, and they're perfectly happy with the deal.

They're not as vocal as the zeitgeist here on HN, and they're not irrational, naive or helpless; they just have a different set of values.

I agree with you that people who care deeply about this issue have the power to change it through focused, collective effort (even if that means supporting open alternatives, like DuckDuckGo). However, I disagree with your characterization of people who willingly support this as being predominantly exploitive.

But we're not talking about them. We're talking about the millions of people who don't want that deal, or wouldn't if they thought about it, but are currently being forced into it by network effects and lack of existing viable alternatives.

I'm talking about them because I disagree with the ideological sentiment that they are not a significant portion (perhaps majority) of the population. How did you quantify the "millions of people" who you claim are ignorant or unwillingly forced into it?

It's easy for people to misunderstand others' values as being based on misconceptions and assume they are either ignorant or malicious. Members of identity groups (i.e. everyone) tend to, in the aggregate, treat their beliefs as positive statements, not normative ones.

Bringing this back to the subject at hand - the point I'm making is relevant because I would postulate that most people make this deal while being reasonably cognizant of the consequences, and simply being okay with it. They don't need to understand how the data is used at a deeply technical level to understand that it can be used in ways that reduce their overall privacy.

I know many non-technical people who can demonstrate an awareness of the way their data is used to impact their privacy in tech products, and they're okay with it. I even know many technical folks who use the typical banter about the NSA, government surveillance and corporate privacy indiscretions ironically, and don't actually care. They just think it's humorous.

You're making the argument from apathy. Sure, there are a ton of people who don't care. But what interest of theirs are you defending? "Don't care either way" is not an objection to change. Ambivalence about privacy is not the same as a preference to give it up.

I'm defending the idea that they are disinterested in change not because they are ignorant, but because they are fully aware and accepting of what they're giving up. I am not arguing that they don't care either way.

If they otherwise don't care, but they appreciate the usability and convenience features of these products, then that is an objection to change if they won't get the same features in more open, private alternatives.

My second point is this: if enough people can be described as both aware and accepting, then mainstream change will not occur because people like what they have more than they like their privacy, and they make that decision in an informed and responsible manner that does not provide a market opportunity.

You can make an alternative, and it can be great (e.g. DuckDuckGo), but it will not be mainstream.

At which point you're in the minority because the set of people who affirmatively want to give up their privacy is not larger than the set who either don't want to or don't care.

DuckDuckGo is not decentralized, it's just a worse Google with a better privacy policy. Actual decentralized search is hard. But this is not a one-time global binary decision. We can have decentralized YouTube and Facebook even if nobody has yet figured out how to have decentralized Google.

I would suggest that having Facebook's userbase is a core part of having Facebook. If you don't have Facebook's userbase on your "decentralized Facebook", you have a strictly inferior replacement to Facebook. The set of people who care enough about the benefits (which do exist!) of decentralization is not and never going to be enough to overcome the downside of having nobody there.

That's the problem you have to solve if this is something you really care about. Haranguing people because they're not using it isn't the way. The way to do it is to figure out how to make it attractive by the users' metrics rather than your own.

> That's the problem you have to solve if this is something you really care about.

We already know that. It's not trivial but there is a clear path.

You make something that will become popular for reasons unrelated to the thing it should replace, it just happens to also be able to replace it. Once you have something that everybody already uses for other reasons and can do everything Facebook does, anyone can start using it instead of Facebook without waiting for anybody else.

The interesting thing is this bootstraps across the whole space if you can do it once. Given a decentralized social graph you can take over for YouTube as soon as you have video support etc.

I would say it's not trivial and there isn't a clear path.

YouTube is not just a web server for videos. You can't set up a decentralized, P2P real-time streaming network (like webtorrent) to replace YouTube. Many of the features that make YouTube virally successful and a continuing platform are developed based on research produced by its analytics which sacrifice privacy to improve insights into what users care about.

YouTube also has a profitable business model, which provides a capitalist incentive to improve the product according to user demand. To make a comparison - which are you betting on in 2016 - IRC or Slack? Even if you set up a successful open source project to replicate YouTube, how will you develop it as well when your best employees are poached because they aren't paid as well as centralized companies?

How about the analytics features? How do you design an anonymous or decentralized recommendation engine with the same or similar utility? How do you power search relevance in a meaningfully private and decentralized way? How do you add the social networking features that people care about without a centralized social network?

The fact that you can technically make product clones with distributed graphs does not on its own constitute a feasible replacement. The basis of modern product research and improvement is data tracking and analysis. You can argue that it can be curtailed or scaled back to improve some privacy, but we haven't even started talking about the performance constraints of decentralized search, streaming, etc. Where will the primary files (e.g. videos) and supplementary files (e.g. recommender models, analysis features, graph, infrastructure, etc.) be hosted? How will you guarantee similar uptime and performance rates to AWS and GCP with a decentralized network of seeders?

You strictly cannot replace products like YouTube, Facebook, Netflix or Google with decentralized counterpoints. The demand and feasibility is not there.

> Many of the features that make YouTube virally successful and a continuing platform are developed based on research produced by its analytics which sacrifice privacy to improve insights into what users care about.

I think you're overselling it. YouTube is successful predominantly because they originally had first mover advantage and now have a direct link from google.com.

The problem decentralized services have is the same problem Google has had in taking over for Facebook. You need to provide something which is enough better than the incumbent that it can overcome the switching cost. The problem has nothing specifically to do with decentralization, and time has proven again and again that it can happen, every time one network replaces another. The next time it happens we just need the new network to be a decentralized one.

> To make a comparison - which are you betting on in 2016 - IRC or Slack?

Here's a different question -- which are you betting on still being used in two decades, IRC or Slack?

That's one of the huge advantages of decentralization. Things stick around because they don't die when one company goes out of business. Which means the process of incremental improvement can refine them into something that, once it has the incumbency, is almost impossible to displace. Examples: TCP, email, Unix/Linux, DNS. None of them are perfect but they'll be with us for the rest of our lives.

> Even if you set up a successful open source project to replicate YouTube, how will you develop it as well when your best employees are poached because they aren't paid as well as centralized companies?

The same way it works for Linux or BSD.

> How about the analytics features? How do you design an anonymous or decentralized recommendation engine with the same or similar utility? How do you power search relevance in a meaningfully private and decentralized way? How do you add the social networking features that people care about without a centralized social network?

Decentralized search and recommendations are hard. So do the easy part first.

Look at YouTube. It's really two independent pieces. One is a web host for videos, the other is a player app with search and recommendations. So you decentralize the hosting and then people make apps with central servers that do recommendations and search. Tomorrow somebody else will figure out how to decentralize the other part. You can make progress without having to solve every problem in the same place at the same time.

> The basis of modern product research and improvement is data tracking and analysis.

The underlying assumption is that you can't ever be finished with anything. But the more time passes the more refined the product becomes and the less you need to fiddle with it anymore.

One minor point - they didn't have first mover advantage.

There were a lot of video sharing sites around before YouTube.

>>At which point you're in the minority because the set of people who affirmatively want to give up their privacy is not larger than the set who either don't want to or don't care.

The set of people who affirmatively want to give up their privacy is in the majority if most people don't care about privacy but want privacy reducing features. Sure, in a perfect world they'd take both, but we're talking about which is more important to them when they have to make a decision. You're arguing my second point, about market viability of a privacy-enhancing product. My first point, which supports my second point, is that most people give up their privacy because they understand the compromise and don't value privacy more than convenience-enhancing, privacy-reducing features.

I'm not sure if I'm conveying my points well, let me try to rephrase. I don't think my postulated demographic is in the majority if they 1) desire competitive features which reduce privacy, 2) are willing to responsibly reduce their privacy for those features and if the market alternatives (decentralized or otherwise) increase their privacy without competitive feature parity.

Basically, if the majority of people are ambivalent about privacy, understand the privacy compromise and still desire privacy-reducing convenience in features, then the alternatives will never become mainstream. This is my hypothesis, and I am pushing this hypothesis by defending the idea that most people give up privacy because they understand but don't value it, not because they are naive or ignorant and value it. The reason I am putting this hypothesis forward is because I think it's important that the HN crowd consider this possibility.

It's easy to believe people give up their privacy because they don't have alternatives, or because they don't know any better, but that perspective comes across as condescending to people who fully understand the compromise they agree to, and there is a strong argument that this is the majority of the population. If that's true, you can't expect privacy-enhancing products to become mainstream if privacy is the only competitive feature, or if the privacy precludes competitive features offered by other companies.

The overall conclusion, based on these premises, is that many privacy-valuing technies misunderstand the privacy values of the majority of the population. They believe that the aforementioned privacy-enhancing products can be competitive and replace the incumbents if only they could get people to understand the privacy breaches they're suffering. My grand unifying theory here begins with the hypothesis that most users actually understand the privacy compromise, and it's really the vocal HN zeitgeist that misunderstands the awareness of most users in this arena. By extension, they also misunderstand the market oppurtunity, mainstream potential or even feasibility of open alternatives that can change or replace the incumbents.

At this juncture I'm not sure how else to convey my point. I feel as though we are probably talking past each other.

OK, so let's actually talk about those people then. The theory is that there is an unavoidable compromise between privacy and features and some people will legitimately choose features. That can be true, e.g. to do "people like you also viewed ..." you need the data on what people like you also viewed.

But those things are relatively rare and can be separated out. You can layer them on top of a decentralized system -- run this separate program that tells The Cloud everything you click on and in exchange you get the benefit of data from other people who make the same choice. That is still possible even if a video from your family gets downloaded P2P directly from your actual friends and no corporation automatically finds out about it and stores it in a database forever.

The problem currently is that the decision of whether to give up privacy is not tied only to the few features that legitimately require giving it up, it's also tied to participating in the same networks as the majority of people and many other features that don't inherently require privacy invasion.

Decentralization doesn't take away the choice whether to share data from the people who want to do that, it gives it back to the people who don't.

The data analysis required to improve search relevance, video recommendation, A/B testing, feature improvement - basically anything, relies on crunching data on as many users as possible. For example, you cannot get to Google or Facebook's level of artificial intelligence research and performance achievements without truly vast amounts of data.

This criticism of decentralization is in addition to my parallel comment in this thread about how you cannot realistically and significantly improve privacy with feature parity.

The performance and reliability would also be difficult to manage without centralized servers. You're not going to be maintaining the same uptime guarantees.

Finally, how are you going to motivate development in a decentralized manner? You're removing capitalist incentives to improve the product. This would have to be managed by a consortium of companies, which doesn't sound like a much better situation than we have now, or it would have to be an open protocol. If companies weren't earning a profit on developing features for further user demand, how would the products improve as well as they do now?

> For example, you cannot get to Google or Facebook's level of artificial intelligence research and performance achievements without truly vast amounts of data.

This argument is self-defeating. If any significant plurality (e.g. a third) of people opt into the "privacy for analysis" setting then you still have "truly vast amounts of data" and there is no trouble. Whereas if so few people want to make that exchange when the choice is made explicit that it can't even work properly then it doesn't matter how well it works because nearly everyone doesn't use it.

> The performance and reliability would also be difficult to manage without centralized servers. You're not going to be maintaining the same uptime guarantees.

Uptime is just math. You decide how much service uptime you need and based on the average device uptime that determines how much redundancy is necessary to achieve it. Consumer devices have lower uptime so you need somewhat more redundancy. And in practice not even much of that, because you want "close copies" (to improve latency/efficiency) anyway, so if you have enough copies for that then a failure doesn't reduce uptime, it just requires you to use a far copy that once.

Moreover, if you have something that really does need specific uptime guarantees or is unusually likely to incur a DDoS, nothing stops you from pinning it to a P2P node hosted on the likes of AWS or CloudFlare. Then to lose uptime you have to lose the entire P2P network and CloudFlare.

> Finally, how are you going to motivate development in a decentralized manner?

Much the same way as we motivate development of Windows, Linux and Wikipedia.

If you depend on P2P to avoid centralized servers, you're just moving the data security issues to devices that are much, much harder to lock down. That can make privacy much worse in a practical sense.

Not to mention the difficulties of reindexing and fast retrieval over P2P. If I need to add an additional index on my centralized servers, the development cycle is much shorter than forcing every peer to reindex itself.

> If you depend on P2P to avoid centralized servers, you're just moving the data security issues to devices that are much, much harder to lock down. That can make privacy much worse in a practical sense.

There is no magic security pixie dust inside a data center. If the server is vulnerable and your data is on the server then you're the same amount of screwed as if the P2P app is vulnerable and your data is on the client. Possibly more screwed because central servers have data for multiple users which give the attackers more incentive to break into them.

> If I need to add an additional index on my centralized servers, the development cycle is much shorter than forcing every peer to reindex itself.

And why is that?

I'm not saying it's magic, I'm saying that securing a P2P network is more difficult because the threat surface is much larger and because you have much less control over your stack. At the very least, I (should) have physical security over my own servers, and the intra-DC data links.

You can't say that at all about a distributed P2P network, so you'll need to put in more elbow grease to address those vectors. For example, if I find a severe vulnerability, I can immediately patch my own servers. If the vulnerability is in the P2P client, it may be impossible to guarantee that every last client gets patched.

> And why is that?

Because there's obviously an inverse tradeoff between control and development speed. Managing distributed state is hard enough when it's on your machines, it's exponentially harder when it's random devices somewhere on the Internet. Your guarantees are much looser.

For example:

1. Your data is less local so there is less effective latency.

2. You're spending computation and bandwidth that other people are paying for

If I need to get some big batch job done quickly and the data is on my servers, then I just up a bunch of instances and get it done. If the data is on a bunch of smartphones somewhere, I can't suddenly grab a bunch more of their computational capacity and bandwidth without pissing off a bunch of users, can I?

And what if I need to change the protocol? We're back to the problem of patching all the clients.

I'm not saying these technical hurdles are insoluble. I'm saying they are real and to date nobody has actually solved them.

> I'm not saying it's magic, I'm saying that securing a P2P network is more difficult because the threat surface is much larger and because you have much less control over your stack. At the very least, I (should) have physical security over my own servers, and the intra-DC data links.

With a centralized system you have this:

client1 <-> central server <-> client2

With a decentralized system you have this:

client1 <-> client 2

In the first case, if either client is compromised then the data is still compromised regardless of what happens at the server, because the data still traverses both clients.

And in the second case the "central server" is better than physically secure, it's non-existent. That method of compromise is removed entirely. There are no intra-DC data links to worry about.

> For example, if I find a severe vulnerability, I can immediately patch my own servers. If the vulnerability is in the P2P client, it may be impossible to guarantee that every last client gets patched.

For serious vulnerabilities the solution to this is to push the update with a date check in it that gives people a reasonable amount of time to update their clients, and after that date all of the updated clients refuse to talk to the unpatched ones.

> Because there's obviously an i, wait a reasonable amount of time for everyone to have instnverse tradeoff between control and development speed. Managing distributed state is hard enough when it's on your machines, it's exponentially harder when it's random devices somewhere on the Internet. Your guarantees are much looser.

Only if you're testing in production. If you're the actual developer then you have a test network which is completely under your control, or maybe an isolated group of beta testers who have chosen to allow you to force-update their machines.

> 1. Your data is less local so there is less effective latency.

With a central server the data is on the server and the client has to fetch it every time it wants to do anything. With decentralization you can keep the data closer to where it will be used, e.g. (a copy of) your photos are already on your device.

> 2. You're spending computation and bandwidth that other people are paying for

You're always spending computation and bandwidth that other people are paying for.

If a decentralized system uses 30 seconds of compute time every day from each of a billion devices, probably nobody even notices (especially if you select for idle devices), but do the same on AWS and your boss is going to want to know why the bill is so high.

> Much the same way as we motivate development of Windows, Linux and Wikipedia.

Isn't one of those things rather unlike the other two?

They're all completely different. Most Wikipedia content is created by volunteers. Most Linux development is funded by corporations the likes of Red Hat and Intel. Most Windows development is done in house by Microsoft.

You obviously mean Windows as the outlier, but decentralized is orthogonal to proprietary. TCP/IP is decentralized but people still make money selling proprietary operating systems with TCP support and proprietary TCP libraries and proprietary routers and so on.

> not because they are ignorant, but because they are fully aware and accepting of what they're giving up.

Do they know what they are giving up?!

> I'm talking about them because I disagree with the ideological sentiment that they are not a significant portion (perhaps majority) of the population. How did you quantify the "millions of people" who you claim are ignorant or unwillingly forced into it?

Yet, when "network effects" drive something to a place considered undesirable, even for a minority at the time, we occasionally pass laws.

See: smoking bans. Network effects (smokers are way more profitable) prevented non-smoking bars from ever forming. So, we passed laws to level the field.

I'm not saying we should do that here, but it is not without precedent.

Usability is a huge factor. I run my own website with Jekyll, but can I recommend that to someone who's not a developer. (After working with/fighting with Jekyll, I'd probably even tell other developers to just write their own).

I'm currently trying to stand up some docker containers for gnusocial and paperwork. Paperwork at least has a dockerfile and docker-compose file in it.

These take a lot of work. There's a huge barrier to entry.

The "People don't understand" is a condescending statement and is exactly the reason why the people who try to build these decentralized systems never succeed, because they don't even understand why people use information silos.

If you really think that's the problem and you think you know better than other people, you should quit using every single centralized system out there, like your bank (they take your money and invest it elsewhere! And most people don't even know about it!), credit card (It's just a small piece of plastic but you have to pay them money just to pay money, and they even track your purchase history, how ridiculous is that!), etc.

Sure there are shitty aspects to this (and I don't like it either), but the reason they are still around is NOT because most people don't know, but because most people have better things to worry about. I know EXACTLY what's going on but still use centralized silos for certain things because it makes sense at the moment, even though there are many downsides.

It is these "decentralized" people who are out of touch, not the ordinary people.

Or maybe both of you are on the opposite sides of a false dichotomy. Just sayin'...

Yes.

E.g. centralized Snapchat has a multi-billion market value based on marketing of ephemeral messaging and a fun UX. Despite technical limitations of the marketing claims, they have reduced the audience for their "historical conversations" from "all customers" to "malicious actors", which is a big step forward.

Snapchat created a multi-billion dollar market via centralized tech and money taken in on the promise they'll monetize it (or the data under EULA) somehow while giving app away for free and operating in a pseudo-police state. I was hesitant to think anything good would come of that after the main SMS and Facebook alternative I pushed for mass audience, WhatApp, was acquired by Facebook in one of largest acquisitions in history. (sighs)

Also, WhatApp was losing incredible amount of money with its number of users despite using efficient, centralized tech. I bet they'd loose even more if people were mostly sending images. I've been thinking secure, decentralized apps with no advertising or acquisition will cost money to develop that will require paying users. Or some clever freemium model. Also, not in a police state. ;)

There are perennial claims that consumers don't want privacy, yet Snapchat used the carrot of ephemeral messaging to gain a large initial audience at the expense of Facebook and other competitors. That data point of market demand for non-persistence is valuable, regardless of Snapchat's motivation or the consequences of their technical implementation.

In the bigger picture, Snapchat can be viewed as Instagram v2, rather than WhatsApp v2. Instagram educated consumers on the use of filter-driven photography as content creation. Snapchat is extending that trajectory, with filter-driven videos being authored by kids in less than a minute. Where might that lead? AR in the style of Google-funded Magic Leap, i.e. animation overlays on filtered reality, which will inevitably be used for advertising, e.g. http://hyper-reality.co. They recently launched "connected" sunglasses.

Have you looked at http://matrix.org? They have telco funding and a long-term vision for OSS and decentralization, so are not subject to typical VC/startup pressures.

"There are perennial claims that consumers don't want privacy"

There's perennial evidence that consumers won't act toward privacy except outside of very, narrow constraints. You're analysis of Instagram to Snapchat is correct. Yet, with the carrot of ephemeral messaging, they went to a centralized service who could keep all their stuff to sell to whoever with no security or assurances in the whole process. Similar with WhatApp in terms of assurances. That's the best we're seeing. Whereas, the usable, truly-private apps that jump into any of these markets get virtually no adoption.

What you're seeing is the power of marketing rather than delivering privacy. Two very different things. Kind of like how many juices market themselves as healthy and natural while actually being a pile of sugar that might harm you if you consume enough. But people drink that 100% juice kool-aid. We've also seen where that went in the corporate space where virtually nothing is actually secure despite marketing claims and the fact that corporations try to hire people that can make an informed decision about that.

Definitely money to be made pressing that button of "ephemeral" on devices that couldn't be that if you tried. Doesn't inspire a lot of confidence for those of us pushing or developing actual offerings that are secure or private. I think more can be done and even maybe widespread but the evidence indicates market will always be dominated by scheming oligopolies of evil companies pretending to protect people with their offerings. Because people don't do better as users or buyers.

No, it's not marketing. That's back to the condescending attitude again.

The people who use Snapchat have a threat model that includes the ordinary people around them: parents, friends, friends of friends, etc. It does not include governments or the provider of Snapchat itself. Given such a threat model it makes perfect sense to use something like Snapchat - it isn't the result of "marketing" (do services like Instagram and Snapchat advertise at all? I've never seen an advert for them).

The reason decentralised crypto-to-the-hilt products never succeed is they misunderstand the threat models ordinary users are using. Most people fear privacy violations by people who know them in some way - not remote faceless programmers an ocean away who, they figure, don't care about them at all.

"The people who use Snapchat have a threat model that includes the ordinary people around them: parents, friends, friends of friends, etc"

Who were downloading apps that could keep Snapchat pics at the time Snapchat users were telling me they went away and nobody could retrieve them. The Snapchat company could also sell them to the same people. That means, due to no attempts to learn about online privacy, they falsely believed that what the marketing told them was true. Or their friends who spread the same propaganda.

So, let's recap. They had a false belief due to a lack of knowledge of and apathy for the most basic information on digital privacy of the devices they ran their whole lives through. They believed these incorrect things about their own threat model because they listened to a marketing team or their friends' repetition of a marketing team without due diligence. Even mainstream media was posting articles citing security professionals saying its claims were false. Snapchat users were spreading knowledge on Facebook that someone could keep the Snapchats. They still stayed on the bandwagon.

Conclusion: it's entirely accurate, not condescending, to say these people didn't know crap about privacy of the apps they were using and/or were too apathetic to research better options. They continued to do so after lies were revealed since the app was fun and popular. They don't care about privacy in practice.

Meanwhile, easy-to-use, private solutions were available and making the media. Hardly any uptake. Speaks for itself.

Making sure nobody can screenshot a snap or save it is an entirely client side programming problem, it's got nothing to do with cryptography, so all the other 'easy to use, private solutions' wouldn't be any different in that respect.

Our point is it's a lie to even tell users that if it's so easy to beat it. Stronger privacy would require a DRMed endpoint where it was a real pain to get those screenshots. Stronger privacy than that is not using the app at all or for anything private. That it's a company whose business is likely to sell out their users is the biggest reason why they don't really get it. Apathy to learning about these fundamental things is what lets such companies get them with false promises.

If you want to say something like this, at least back it up with something. Anyone can say "You two are both wrong" and sound smart.

Also which part of "people don't understand" is not condescending?

Funny enough, I watched this movie called Barton Fink last night, and it's a perfect depiction of this nonsense.

> Also which part of "people don't understand" is not condescending?

How is asserting a fact like that in any way condescending?

Because it's not a fact

It is a fact. Some people don't understand. And even if it were false, that doesn't make it condescending. People are wrong about things all the time without it being condescending.

Go back and re-read the original comment I replied to. You're just arguing for the sake of arguing by taking the words out of context.

> You're just arguing for the sake of arguing by taking the words out of context.

No... I specifically asked because I wanted to know why you thought it was condescending. That's a far cry from arguing for the sake of arguing. And it didn't come off at condescending to me.

Maybe you think the web can only be totally decentralized or totally centralized, but I don't think so and that's why I made the comment. Someone can be fine with centralized banks and still favor decentralized social networks and running your own p2p servers, for instance. If you disagree, fine, but don't pretend that my post was condescending.

Take it easy.

I don't know where you got the idea that I think the world can be totally centralized or decentralized, honestly.

The bank analogy was about how there are upsides and downsides to everything, and the society is not as stupid as to have no clue what's going on. For things like this the world follows the collective intelligence.

Even the people who have no idea how a bank works still use bank with no problem because of the collective knowledge. We know that a bank is safe enough and provides more upsides than downsides.

But this is not always the case. In countries where people have low confidence in their government, even the "ignorant people" who have no idea how Bitcoin works learn about the benefits and move away to it. This is not a result of some intellects trying to convince the "uninformed" how the world should work. It's a result of the collective intelligence recognizing how the pendulum has swung and now there's more downsides than upsides.

1 . > It is these "decentralized" people who are out of touch, not the ordinary people.

This statement seems a bit condescending, just saying..

2. In my post I'm not trying to make value judgements, but just offer a possible explanation why things are the way they are, and who may have better chances to change them.

3. I don't think that we should/can decentralize everything, but I do believe there is a worrying tendency towards extreme centralization. This may not neccessarily be a bad thing though. The world we live in is the expression of our choices, and if this is the result, then this is the result. We can observe and adjust if we feel the need, by changing our behaviour, if not, then that's that.

> 1 . > It is these "decentralized" people who are out of touch, not the ordinary people. > This statement seems a bit condescending, just saying..

I don't think that's what condescending means.

Decentralization/Centralization is nothing new. In fact the entire history of humanity is about pendulum swinging back and forth between centralization and decentralization. And in 100% of the cases when this happens it's not because of some elites convince people to switch, but because it makes sense for people. "Normal" human beings are not as dumb as these elites think. So we don't have to worry about decentralization never happening. It will happen. And probably back to centralization. And then back to decentralization again. Just not through these efforts to "educate" people. It's because these elites are so out of touch with how real people think that their attempts at convincing people don't work.

> I don't think that's what condescending means.

Not a native speaker, so maybe I got that wrong, but this is the definition Google gives: "having or showing an attitude of patronizing superiority.", and this is what I was referring to.

Agree with what you say about the varying degrees of centralization/decentralization throughout history, but don't think that these changes happen out of the blue. Education is usually needed to bring about change and not because people are dumb, but because information doesn't always gets to them easy. So maybe this is what's happening now. You can think of it as a balancing force...

>Not a native speaker, so maybe I got that wrong, but this is the definition Google gives: "having or showing an attitude of patronizing superiority.", and this is what I was referring to.

Well, who is patronized? Those decentralized groups that think that the masses "don't know what's best for them", or someone pointing out that such groups are out of touch with reality?

In the end, any critique can be though as "patronizing" -- it implies the one making it knows better than those he criticizes. But I think that's too low a barrier.

>This statement seems a bit condescending, just saying..

It's usually a safer bet to dismiss the assumptions of some small group of people than think that the whole general population are idiots.

Just saying...

http://i.imgur.com/oTBYE5r.jpg

I agree, it's usually a small group of idiots that conflate high IQ with not being an idiot.

If that was really true, it's be a safer bet to dismiss what experts in a given field think and just go with what the uninformed population's opinion is on the matter.

E.g. to dismiss scientific opinion on a topic to go instead with lay opinions on it

By all means, we should dismiss what experts in a given field think too.

What we should respect are hard results and proofs from said experts.

Science is not about trusting experts, it's about trusting a methodology (the scientific method) to produce results.

Yes, but I was responding to a claim that equated to "go with the most broadly held view".

It's a tragedy of the commons situation.

Even if you understand the downside of centralized systems (and most people really do not), you cannot individually do anything to stop them. So you use them out of convenience.

"Decentralized" people are out of touch only to the degree that they believe that they can change the world based on collective altruism. That is an uphill battle. They are not out of touch in realizing the dystopia that we are heading towards.

I imagine two possible scenarios for decentralized systems to take over:

a) Dystopia goes mainstream -- one scenario: "social credit scores" become common practice and people start to fear using the web to an extreme degree;

b) Someone comes up with a decentralized system that is able to do something, only possible with decentralization, that everyone really really wants.

I hope b) happens first.

People would change the way they use the web, not stop using it.

Yes, that is precisely my argument. Make the normal web sufficiently oppressive, and the freedom provided by decentralization becomes so compelling that people switch to it.

Do you seriously expect that in dystopian scenario altenative going to be available? Even if decentralized services going to remain available do you think people will risk using them?

Most likely they won't going to do that. This is proved by China and more extreme example of North Korea also available.

Or just look at any country with totalitarian regime: they increase regulations really slowly so people "too busy" to think about that, but at some point they simply too scared to do anything.

You have a point. What I imagine is that some underground "resistance" would develop and use such systems, and once the authoritarianism eventually collapses, the mainstream will then be afraid of centralization.

Take as an example the Germans' insistence on privacy, to extents that even seem ridiculous to a lot of people (e.g. the right to blur your house in Google maps), and then consider their historical context.

China would seem to prove the opposite: extensive controls are in place, yet many people find ways around those controls.

Reportedly teenagers, who are much more aware of networks' downsides by having grown up with them, are trying harder and harder to turn off or at least reduce their network footprint. See for example the rise of ephemeral services like Snapchat, 4chan, pastebin and so on.

(B) is what Bitcoin did with drugs, assassination, etc. and BitTorrent did with file pirating.

That said I agree with you and that was exactly my point. Instead of coming from altruism they should build something genuinely useful that's only possible as a decentralized system.

> assassination

Do you have any evidence that this ever happened?

First, I am speaking from my own browsing experience on the now-dead silkroad.

Second, I'm not trying to disagree with you at all. It was to point out that there are definitely use cases for decentralized technologies where centralized systems can't go (although a lot have been negative so far)

I know you are not trying to disagree -- and if you were, that would also be fine, of course :)

Mine was not a rhetorical question. People keep mentioning assassination markets but it always sounds like a urban myth to me. I would like to know if there are documented cases or not.

Just because you see an advertisement on something like silk road, doesn't mean that it's not some kid playing pretend.

idk how legitimate they are, but hidden wiki does have entries for forums for this.

Trying to buy a hitman was how they caught the creator of the silk road.

It's possible to deploy SaaS software on-prem now. There's a c) solution that takes the middle path where we all win. Unfortunately it takes implementing a type of "suffering currency" to enable how people pay for it. If they are willing to slow down a little for reliability, they get more reliability. If they insist on running recursive workloads, and in general just chew up a bunch of resources for no gain, they get their workloads slowed a little. Works with the Open Source model too.

"The "People don't understand" is a condescending statement and is exactly the reason why the people who try to build these decentralized systems never succeed, because they don't even understand why people use information silos."

I've talked to at least hundreds, maybe a thousand, people (not techies) about these issues over past 10 years. Most don't understand the scope of the problem in terms of the tradeoff they're making. Many also think main suppliers couldn't, wouldn't, etc do sinister things X, Y, and Z because something would happen. Many think problems X, Y, and Z are inevitable since all the famous apps allow them for extra profit or growth. Vast majority don't know much about computers, privacy law, or U.S. history of surveillance states (eg Hoover) because they don't put time into learning. They will shout about minor things that are similar on TV they understand with little thought.

So, it's not condescending. It's a fact that "people don't understand," are apathetic, or terrible at long-term risk/reward tradeoffs. There's shitloads of empirical evidence confirming it in terms of what they adopt in the market vs what problems it causes them. And what crap they adopt next time. We definitely should put the share of blame on them that they earn.

"It is these "decentralized" people who are out of touch, not the ordinary people."

A lot of them are, too, in fairness. They come up with all kinds of crazy, unrealistic solutions that ignore human nature, economics, etc. Best models are going to be hybrids.

> It is these "decentralized" people who are out of touch, not the ordinary people.

There are plenty of countries in the world where "ordinary people" and "political active people" are two different groups.

Ordinary people there have better things to worry about than politics. Many of them also know what exactly going, but do nothing to prevent it. Usually this end up really really bad.

> There are plenty of countries in the world where "ordinary people" and "political active people" are two different groups.

These "ordinary people" often end up allowing "extraordinary" things to happen to these "political active people".

> It is these "decentralized" people who are out of touch, not the ordinary people.

Exactly. I'm still not sure how we ended up in this world where people no longer understand market competition and consumer value choices. People choose platforms and products based upon what is convenient for them, not necessarily best for them. In the 90s AOL became the internet for many people because it met their wants and needs, much like Facebook has today. Consumers/users are not idealists with regard to their choices and they never will be.

> but unless they make the choice to listen and understand it's all for nothing

And when do we usually listen? When we feel the pain. Which is why we'll probably have to feel the pain of authoritarian governments in the near future, to gain enough understanding, before we can make that transition to a better humanity. I just hope this painful transition won't take too long.

We're all choosing centralised silos despite having personal preferences otherwise. This is a systems outcome, and not one we'll change individually.

A distributed social network may arise, we just haven't yet seen the initial conditions, value chains, and systems structures that will bring it about.

This article gets excited about blockchain cryptocurrencies as the substrate, since money is a ubiquitous need. Well, it's worth pursuing. I'm still angry with Google & Facebook for killing off federated chat, which was the other nebula in which a new star protocol might've been born.

You are underestimating the power of external influences. It's not just about what individuals decide for themselves. Because right now, even if you decide to learn what it takes to have your decentralised internet back, that still may not be enough.

We still don't have an easy to use computer that can run 24/7 in your own home, on which you can install a blog, vlog, social stuff, email, as easily as you use the current centralised services. When I first installed my mail server, it took me a week. I'm not a system administrator, but I am a developer familiar with GNU/Linux. Your average muggle doesn't stand a chance.

Even if we had those little computers, our bandwidth is still asymmetric (even the fibre, even though it makes no technical sense) and that decision is a corporate one, that you are not going to reverse by voting with your wallet.

Even if we had our symmetric bandwidth, we still cannot send emails from our own home. I mean we could send it, but it would be rejected by every big players out there, as well as most mail filtering systems, because the current assumption is that if it comes from a residential IP, it's spam.

And we still don't know how to make a decentralised search engine.

---

The choice these days is simple: either you give up on the likes of privacy and anonymity, or you shut yourself from the network. It's not a binary choice —more like a spectrum in fact— but there is no third alternative yet. For instance, I don't use Facebook. I missed some events because of that, and people are surprised I didn't know about that party I was invited in. It's the fifth time I told them I never log in, dammit!

and then there's still the data caps :(

Not if you live in a developed country... ;)

And that brings the circle back 'round to where it started (again). SIDE (but related) NOTE:Check out the book "Throwing Rocks at the Google Bus."

You know, I've come to the same conclusion from a different direction. I decided that people's expectations frequently exceed the cost of fulfilling those expectations because they spend a lot of time speaking for others. All of this stuff we're doing right now is non-scalable when it comes to implementing everyone's different privacy concerns at scale in a centralized solution. We simply can't scale infrastructure at the speed the public expects, while at the same time throwing away privacy rights it would take them half a lifetime to understand.

I expect eventually that with the rising tide of machine learning solutions being developed, someone will figure out that the chunky coarse-grained predictions and latency-filled transactions between user and server is monumentally inferior to a segmented approach which leaves the vast majority of user preference information on the clientside and which performs last-step filtering on the client device. No company can afford to throw the equivalent of a smartphone in processing power (plus what is necessary to make up for the latency) at each and every single users query, so purely server-side solutions will always be worse. Some day, someone will deploy a hybrid and it'll be faster and slicker than anything the companies desperately pimping their customers personal data can manage. And they'll sell it as a 99 cent app, not a subscription service or an indentured servitude "free" contract. And then, maybe things will change a little.

A real barrier to a decentralised web is the difficulty of installing software on a server. I know that sounds really mundane and inconsequential in the broader debate about a decentralised web, but consider the following...

Imagine if installing a server-side chat app, message board, project management app, or CMS were as easy as installing a desktop app. In a desktop app, it's usually one click to start the install and then, if necessary, you're guided through a few screens to complete the install. Want to uninstall? The OS (operating system) will provide a feature to manage that.

Now consider how complicated installing on a server is in contrast. Upload you files to a folder or directory, enable permissions, set configurations not just for your server but also the language the program is written in - the list goes on. No wonder SaaS (Software as a Service) is thriving like never before. Who, other than technical folks, could possibly have the time, interest or inclination to set up a self-hosted solution when the barrier is so high? Perhaps some in the tech field would like to keep it that way? Would Saas be less attractive if installing a self-hosted solution was simple, easy, quick and secure?

Surely an essential part of a decentralised web is that companies, organisations and individuals choose to run their own software using open protocols and data formats. But until the ease, security and simplicity of installation improves for web software, it simply won't happen on a large scale.

Living billboard arrives to announce, drumroll please :

sandstorm.io

The auto updating server that you can self-host or use as a hosted service, allowing you to install and uninstall lots of applications, both free and proprietary, just by clicking the button.

(No, not paid to tell this unless you count the stickers I got. And no, not perfect but still a refreshing change.)

Sandstorm is awesome. Of course, it would help if ISP's weren't so keen to make sure nobody actually uses the internet to make computers talk to each other, instead of just having Amazon, FB, and Google's computers talk at you.

ISPs don't do that. That's your NAT/firewall.

Has anyone ever actually seen an ISP network that filters local traffic to other ISP endpoints? I'm not even sure why they would do that. DDOS almost always targets big things in the cloud so it would not help there.

Here (SoCal) traceroute to a host in the same local neighborhood is three hops.

There is a highlight point that many internet services are indeed asymmetrical. My Comcast service is 180/20 Mbps, down/up respectively. Even if you get Comcast's new gigabit service, you get 940 down, and 40 up. And many home ISPs prohibit you from serving a website at home, according to their terms of service.

Comcast has symmetrical 2gbps fiber.

The 940/40 is DOCSIS 3.1

The situation with uploads on home connections is better than it was a few years ago though.

It's pretty common to have outbound traffic to port 25 blocked as an anti-spambot measure. Sometimes you can call them and get them to remove the block, other times it's non-negotiable.

Most cloud servers do this too. Spam has basically destroyed smtp as an open federated protocol anyway. Network reachability is the least of the issues you'll face trying to run your own mail.

Lesson: any distributed or federated protocol that is not robust against abuse is doomed.

> DDOS almost always targets big things in the cloud

That isn't true. Those are the ones that you'll hear about in the media but there are plenty of DDOS attacks on smaller sites both for extortion purposes and to force them offline.

That may be true, but in any case the DDOS usually arises from outside the ISP's network so forbidding local traffic would have little effect.

A lot of ISPs have (probably unenforcable) clauses in their ToS saying your can't run services.

Its only purpose is to force businesses up to business class. You might see it enforced if you ran a public for profit site from a consumer connection.

In any case the encrypted p2p protocols run by dapps (decentralized apps) are opaque, and these don't really qualify as servers because architecturally they are not client-server at all.

I can attest to that, my previous ISP had explicitly stated that running web or other file server was not allowed - I'm not exactly sure if TOS was in violation of my consumer rights but the fact is that they could cut me off if I were going to set up a server... I mean even for personal use (it didn't say anything about running a server business)

This. I am an ISP owner and we block popular ports used to run services by default. 99% doesn't care and their computer gets hacked or they get virus/malware and those malware starts running services and spamming the world and our IPs gets blacklisted (which is a huge headache) and now cloudflare's broken shit ip checker will automatically start blocking ips, and customers will start blaming us for it. Occasionally, 1 or 2 customers will ask to have email server port or web server port to be opened, we usually ask them to sign up for a business contract, which clearly states that you can run this kind of services and you are fully responsible for any negative consequences.

This is not about money. This is just good housekeeping. When I was in NY, I believe they TimeWarner and Verizon (or was it AT&T?) also used to do the same thing.

Thanks for responding with the ISP's point of view. My frustration came from just wanting to run a small Ghost blog on an odroid with, at most, hundreds of visitors per month, and hearing that this required a business contract. Of course, whether I want to do this for business or pleasure doesn't really matter if you're dealing with malware and being blacklisted.

At one time ISP's gave you a bit of space on a shared host for free as a courtesy, it would be nice if these days you could get the equivalent of a t2.micro. I know a few people (teenagers I'm related to mostly) who would like to mess around with programming and building web sites but for whom even a tiny cost is a barrier. As it is I throw a few bucks at nearlyfreespeech for them and it's good for a year, but I imagine plenty don't have that option.

I think there is a miss communication between consumers and internet providers. I think if the consumers were aware of how things are run in an ISP they will be more understanding and reasonable when facing a problem like not being able to host services yourself. From my part, I try to write about my struggles in running our ISP business <1>.

Another thing to consider why we can't easily allow our user to run services if we wanted to because most home users IPs are not static, and often your IP is shared by many users. IPv4 is running out, giving our user the ability to run service will also mean giving him a unique IP, which they will have to pay for it and because of the scarcity of unique IPv4s only business users are allowed to have them and business accounts cost money. We haven't moved to ipv6 yet, it will require some investment and overhauling some of our networks. Depending on your local law, you may actually have to have a business under your name, meaning you have to show business license papers, to get a business connectivity.

I know what you mean by having to run your own blog or services. I have done it myself and I have learned what I know today mostly due to me tinkering and optimizing apache and WordPress on the fly when it was on the homepage of Digg. It was such a rush and I have come a long way since those days, thanks to being able to host my blog on a time warner DSL line. This was in the early or mid-2000. But it was a different time, and things have changed a lot since then.

One of the reasons ISPs back then used to give you free email address and shared hosting, because they wanted to tie you down to their services. Specifically, if you used their email for a long time it would be difficult for you to leave it behind. But the landscape has changed since then, consumers are smarter and webmails are much better and free blog hosting are dime a dozen.

My advice to people who want to learn hosting services by doing it themselves on their own computer that, even though it can be a thrilling experience, hosting it remotely and configure/securing server from scratch can be an even better learning experience. And VPS hosting services like digital ocean can be very affordable.

There is a popular misconception that ISPs are a profitable business and they try to shaft the consumers at every opportunity they get. Maybe for the big players like Comcast, it is true, but the rest of us small and medium sized ISP its far from the truth. The complexity in running reliable services can be staggering, the equipment and licensing can be an astronomical investment (at least for us it's a large amount) and running the last mile and upkeep for it with 24/7 support are very expensive for us. BWs are not expensive, but equipment is. When we saturate our 10gbps port, moving to 40gbps or even 100gbps network (switches and routers) can be a mind-numbing expense. On top of it, we are taxed and need high license fees for all kinds of services. We really don't make a lot of money and you constantly have to spend on your infrastructure as you grow (fighting to get customers by lowering your cost, and hence your revenue and profit). You have to keep growing to stay relevant and reach that point you can connect more users with minimum costs without having to invest in growing your network. Reaching that point can take anywhere for 5 to 10 years, if not more.

Sorry for the rant, I just needed to get it out of my system. :)

<1> http://www.slashgeek.net/starting-isp-really-hard-dont/

my ISP blacklists some random ports on consumer contracts.

you could of course go with non-standard ones, its still another complication that they throw your way.

I blacklist 25 out except from our mail server. That's just basic common sense given some of the spyware. Users who need to send mail will use encrypted 465 or 587 anyway.

Wow! Not heard of sandstorm before and seriously impressed!

Have you looked into Sandstorm? Their whole goal is to make installing server apps as easy as in mobile platforms, while securing them from the outside and each other, and allowing the user to switch providers bringing their apps and data along.

It's really quite awesome: https://sandstorm.io/

Here's something from the first part of the install which is not something a normal person would ever do:

Configure the EC2 security groups (firewall) to let you reach Sandstorm

By default, Amazon EC2 protects your virtual machine from being reached over the Internet.

In the above tutorial, we allow the virtual machine to be reached on port 80. By default, Sandstorm uses port 6080, so look through the above tutorial and add another security groups rule allowing port 6080.

SSH in, and run the Sandstorm install script

Uh, if you can't figure that out from the directions, maybe you shouldn't run your own server?

It's ludicrous to suggest that everyone should run their own server. It's like saying they should all build their own cars.

It's more like suggesting that they should drive their own cars, pump their own gas, or change their own tires in an emergency. Nobody is suggesting that people should be able to program a webserver.

I agree, but people aren't going to pay someone to run a server for them when Facebook, twitter, snapchat, etc are free.

It's also hard to argue that it's fundamentally different if you don't run it yourself.

That's a fair point. Maybe we should lobby VPS providers to have simple installers for it, like DO's one-click apps.

I've just installed Sandstorm to test it: Created a droplet in Digital Ocean, logged in ("ssh root@...", and they had my ssh key), the curl|bash, and it worked. It was way beyond awesome for an open-source Linux app. So the EC2 problem exists only because someone advised to use Amazon for a hobby installation.

I love the concept, and I'm looking forward the day a facebook alternative could be built into it. And if it had existed when Google Reader ditched the internet, everybody would be on Sandstorm today. I've just experienced the feeling of browsing my RSS feeds of porn without feeling watched and without being afraid of hitting a "share to facebook" button: It's a great experience.

> A real barrier to a decentralised web is the difficulty of installing software on a server.

Also, economies of scale.

If people used a decentralized service called "UnFaceBook", the total cost of servers, administration, etc. would dwarf the cost of Facebook running their data centres. From a business perspective, it's just not feasible.

Hmm... perhaps everyone running their own systems is, in fact, doable. Most people have smart phones which are much more powerful than servers from 6 years ago. Why not just use that?

Have the content at the edge, and controlled at the edge. Scalability can come from lots of caching at the core.

Not a startup I'd want to do, but it's technically feasible.

I like the idea of smartphones as servers feeding content to a CDN... Combined with IPFS [1] that should work well.

I was going to make a different comment though. You suggest that, because it would have higher infrastructure costs, a decentralized network is "not feasible" from a business perspective.

I'm wondering a) whether infrastructure costs are currently a limiting factor in the growth of social networks, and b) if a decentralized social network needs to be a business at all.

But more to point a, what if it cost 10x more, but the current costs were $0.10 per user per year. Do you think a service with a cost of $1 per user per year would be too expensive to operate?

[1] https://ipfs.io/

I was exactly just thinking about this. If phone is offline we need CDN's. But then again phone's have to watch out for their precious battery life.

With decentralized, you also have a huge issue of protocol. Facebook can upgrade millions of people instantly to new version. Decentralized could be a major pain.

>Most people have smart phones which are much more powerful than servers from 6 years ago. Why not just use that?

Not before we have breakthrough in power cells. Or some other breakthrough. Perhaps computer implanted in our body and powered by our food.

Economics and scale can be a strange beast. The sum of Amazon EC2 + Google Cloud + Digital Ocean + Rackspace +... is about 10,000,000 servers, which makes... 1 server per 7,000 inhabitants on Earth. Have you ever looked at it this way?

And that's only for public cloud, not including Facebook, Google's internal servers, Apple's infrastructure, ISPs, and servers hosted by all companies. So to provide all IT services to citizen of modern economies, we're certainly close to 1 server for 100 inhabitants. Sometimes I wonder what we're doing with so many servers on Earth: I don't spend 24hrs a day sending requests to public servers, and even if I did, the server I'd be pinging could handle a few thousand users at the same time. So where does all this processing power go?

And there's even more computing available if you include everyone's home and work PC, phone and router, but those are not always-on.

> Hmmm... perhaps everyone running their own system is, in fact, doable.

Crunching the numbers, we're already above one system per person ;) So we might as well go full-decentralized, if we could conceive a theoretical model around it.

Certain things, like filtering out spam, or handling video, require rather long computation per user per day. Same probably applies to just transferring data quickly enough, with a lot if spare capacity to handle spikes.

> So where does all this processing power go?

Security, redundancy and isolation. Often times you have an extra server not because you need the processing power, but to separate things for security reasons, to provide failover and to avoid noisy neighbors.

> Most people have smart phones which are much more powerful than servers from 6 years ago.

This isn't even remotely true.

The most powerful servers from 6 years ago is a xeon x7560, which has a 40% higher passmark score than even the best enthusiast consumer-level cpu on the market today (an i7 6700k), never mind even the most expensive smartphone on the market.

https://cpubenchmark.net/cpu.php?cpu=Intel+Xeon+X7560+%40+2....

https://www.cpubenchmark.net/cpu.php?cpu=Intel+Core+i7-6700K...

I think that qualifies as "remotely true" actually. They didn't say "more powerful than the most powerful servers" they said "more powerful than servers".

People on the internet have an odd tendency to interpret statements in the broadest possible way.

I did misread that. I think the point still stands. The cheapest server cpus from 2010 are 60% as good as the best 2016 cpus. It might be close to the iPhone 7's latest chip or Google's Pixel. But most people don't have the latest and greatest. A normal phone won't be close to the latest and greatest i7. And I don't think it will be close to a 2010 era server.

> Who, other than technical folks, could possibly have the time, interest or inclination to set up a self-hosted solution when the barrier is so high?

Some evolution of docker swarm could greatly simplify this.

Or an organisation people could freely join without being forced to learn system administration concepts and practices. Imagine WhatsApp, but owned and sustained by (a subset of) its users on a non-commercial basis.

It is as easy as installing a desktop app. Sometimes even easier (you can not just write "apt-get install Word<Enter>" and it will be just there).

The problem is that you don't just want a CMS, you want it to look a special way, do special things with it, don't want other people to access it without you allowing it, etc.

I dunno, there are an awful lot of server apps that I can install by simply running apt-get install. Achieving nirvana might be a simple matter of packaging.

A web server is not that hard to set up. I personally have one on my local network. I'm sure many people do.

It seems to me that a more difficult part is to make this server public, that is with a fixed IP or domain name. I've never quite went this far.

That should be fixed with IPv6, no need for NAT at that point, and home routers could perform simple DNS functions.

See cloudron.io. they make it easy to self-host.

P2P filesharing is part of the "decentralised web", and people seem to have no trouble installing the software for those.

I think a lot of the difficulty is artificial, created by software that is far more complex than it needs to be, to cover far more use cases than most users actually need. In the "enterprise" space, a lot of this complexity probably also drives auxillary revenue in the form of training, consultancy, etc. In other words, it could be a deliberate barrier to entry. Building big, complex, immensely flexible, yet difficult-to-configure systems with plenty of dependencies just seems to be the norm.

Perhaps some in the tech field would like to keep it that way? Would Saas be less attractive if installing a self-hosted solution was simple, easy, quick and secure?

Indeed, the whole category of "enterprise software" often fits this business model.

But me and many others have written HTTP and FTP servers which do not require any installation at all --- they're just a single (often very tiny compared to most other software) binary, sometimes with an optional configuration file. If you're doing something like hosting static pages, this fits the use-case perfectly well.

Finally, a huge part of making the web truly decentralised is to abandon the notion of dedicated servers/clients altogether --- and thus also the notion that you must need a dedicated or "server" computer to host anything, or for that matter a dedicated Internet connection. Of course some machines will have more resources to serve, and a typical residential connection may be more limited, but the key idea as exemplified by P2P is that any machine can serve.

>P2P filesharing is part of the "decentralised web", and people seem to have no trouble installing the software for those.

We can't really call P2P filesharing today decentralized. It depends on trackers. And it's trackers who help users (often out of commercial interest) overcome all the troubles with software.

Some torrents depend on trackers, yes. Others use magnet links, which aren't dependent on trackers at all. The data typically served by a tracker is instead stored in the DHT:

https://en.wikipedia.org/wiki/Mainline_DHT

DHT helps with downloading, but content discovery depends on tracker. You can't search for, rate or review content in DHT.

Content discovery can happen over any number of decentralized channels including email, diaspora, etc.

Every peer to peer system requires bootstrapping, often this is achieved with some central or federated nodes/trackers. Once you have peers, you can forget about trackers and use the DHT.

> Imagine if installing a server-side chat app, message board, project management app, or CMS were as easy as installing a desktop app. In a desktop app, it's usually one click to start the install and then, if necessary, you're guided through a few screens to complete the install. Want to uninstall? The OS (operating system) will provide a feature to manage that.

It's this way already. Virtually every shared hosting provider (Hostgator, Bluehost, etc.) provides a Cpanel admin panel with Softaculous software installer. All kinds of apps (blogs, CMSs, project management apps, etc.) are a single-click install and removal.

> A real barrier to a decentralised web is the difficulty of installing software on a server … Imagine if installing a server-side chat app, message board, project management app, or CMS were as easy as installing a desktop app.

If you have a desktop, you can install any server software you want. And if you have a desktop, you can leave it running as long as you want. You don't need to purchase a host somewhere; you can just use the computer you own.

Heck, you own a computer which is on 24/7 already: your cell phone. And software written well could run on that phone to serve whatever you want, at a minimal cost in CPU & hence battery.

Software written well can work that way on Android, but not on iOS which force-suspends your process and sockets all the time. The only exceptions are granted unilaterally by Apple and the chance of them allowing server-like behavior on iPhones is infinitesimally small.

That's why it's important to hold on to people owning general-purpose computers, open standards and community governance: because once you move away to centrally-controlled appliances, there won't be a platform from which to bootstrap the next, better system.

> A real barrier to a decentralised web is the difficulty of installing software on a server.

LOL.

Decentralised would mean the user installs software on his/her desktop/phone and it works as you think a server works. Thats all there is to it.

If you make "installing on a server" - somebody else computer easier, its not gonna be detentralised. Youll have shareing of peoples "apps" on a single server, and eventually Cloud is going to be invented. Look then how easy it is to run apps and install shit on other peoples computers.

Running "a server" is not any more difficult than running any other app, on Android or on Linux, the difference is pacman -S kwrite or pacman -S lighttpd, or picking "primitive ftpd" app and running it.

I think the real problem is that people think servers are some kind of magical computers, different from any other general purpose computers people are already using as clients.

> Running "a server" is not any more difficult than running any other app, on Android or on Linux, the difference is pacman -S kwrite or pacman -S lighttpd, or picking "primitive ftpd" app and running it.

Which directory does the ftpd app store it's data in, and on which partition? How do I get alerted if I'm running low on space and what do I do if I am? Is the data backed up and what is my disaster recovery process? What port is it running on and how do I connect to it? BTW, insta-fail because we should be using secure ftp. Is it using encrypted communications and how do I install a certificate and share public keys? What about configuring access through the firewall? etc, etc.

> Imagine if installing a server-side chat app, message board, project management app, or CMS were as easy as installing a desktop app.

Several NAS vendors already offer that, so you can install such server software on their devices with a single click and easy configuration.

To most people (99% ?) Facebook, Instagram, Google etc offer everything they want. So they do not feel a need for getting the 'power back'.

It all comes down to perceived (!) value. Once people use one thing, they will only switch if something else offers a clear higher value. At that point offering the same value is not enough anymore. This higher value has to be something that makes every day life in a very obvious way a little better. I am afraid something abstract like 'more power' will not do the trick.

That also explains to libertarians why people like "The State". They don't necessarily feel they need to decentralize everything. But the real reason is, it just hasn't become convenient yet. One day, each community will run its own open-source software that will manage local taxes and services without having to wait for Big Government to do it.

The trend in private (as in commercial, non profit etc) internet is exactly opposite. Why do you expect the public (as in state) to be different?

Because as I said, trends are largely a function of available technology. There is a reason we put up with long DMV lines and mailed everything with USPS, until the web eventually moves all that stuff online.

The public zeitgeist is finally catching up with the realisation that being entirely dependent on massive siloed community platforms is not entirely in the users’ best interests

I see zero evidence this happening.

If anything I see the public and businesses going the other way toward the cloud to avoid the hassle of IT. For the average person a PC is a broken piece of time consuming junk that accumulates malware.

Edit: There's another comment in this thread where someone brings up the hassle of IT as a major centralizing force. Best comment so far in this thread.

I see it in some places. For example, my sister now avoids facebook and instagram because she doesn't want pictures of her kids on those services. Unfortunately, none of the decentralized and distributed services are easy enough for her and the rest of the family to use without help.

And just because it's decentralized may not necessarily solve the problem of pictures of kids on those services. That problem could still be solved with a centralized photo sharing platform.

Indeed people have been saying this since AOL. This is a tired argument. Nobody is forced to use anything and you can still buy a domain and run your own Wordpress in like 15 min. Hell you can set up your own social network on Wordpress on your own domain in like 15 min.

Many computer users can do this, but most computer users can't. So most users don't have that choice.

Saying they don't have a choice is sorta disingenuous. Almost everyone was born with the required organs. Most of the tech has free alternatives and instruction exists in numerous freely accessible and easily digestible blog posts and YouTube videos. Saying the users have no choice in this matter is like saying a lot of users can't choose to vote because they're not registered and they don't know where the polls are.

Well, I certainly disagree. Technically you are correct. In a vacuum any person, even my own mother, could endeavor to learn these things and do them.

But when you put a person's life in to context, how do you expect a working mother of three to just learn the tools? What if she has the remotest of computer literacy? That is, she could operate Facebook but no more? Then is it reasonable to expect her to learn how to administer her own server? Can she choose to spend hours a week learning for months while her children need food to be prepared for them, clothes to be washed, and beds to be made? In a practical, not theoretical sense, does she really have that choice?

The example with voting feels disingenuous because voting is specifically designed to be accessible to every person. Learning bash and administering your own web server is absolutely not the same.

> What if she has the remotest of computer literacy? That is, she could operate Facebook but no more? Then is it reasonable to expect her to learn how to administer her own server?

She wouldn't have to, it's all about the money and the wish to be dependent or independent. There are web hosting as well as consulting companies that can setup private social network for one time fee, with automated backups and all.

To me, when I hear that some "computer illiterate" person "learns to use facebok" it kinda sounds reminiscent of hearing that someone learns to make crack from cocaine. Is it useful - maybe, is it good for society - eh I don't think so. I rather see them "learn" how to use sandstorm...

It requires time to even learn about how to search for a web hosting company.

Just the words "web hosting" are not in most peoples' vocabulary.

You are right that it's good to teach people computer literacy. But your belief that people can somehow spontaneously learn about web hosting is unrealistic.

People first need to experience something, then get some language for what it is and how it works, and only then can they begin to seek out education about how to learn how to reconstruct it for themselves.

Providing turnkey decentralized application hosting is a way to allow people to first experience what it's like to deploy an application. By having that experience and doing it over and over they can start to have shared experiences around it, from which they will learn language about what it is. Only at that point can they start to learn how it actually works and what it might be like to do it in a different way.

Where the change will come is when we are no longer dependent on paying for network access. That will be the financial incentive.

A fully decentralized web would be delivered peer-to-peer via a mesh network or something similar. Anything else is a farce, because it's not just about who holds the data on the network, it's about the network itself. If any link in the chain of the communication can be controlled by some centralized power- it's not decentralized.

On a fully decentralized network, three things restricting freedom and privacy would be:

* the personal device used to (inter)connect, like malicious code hiding in firmware

* those controlling the power needed for the device

* those that can interfere with communication or alter data on the distributed network, either as a peer on the network, through malware/disruptive communication on the network, or those blocking communication

The problem with the decentralized web, though, is that when the web is fully free- if everyone stores part of the content from everyone else, then they could be storing things that are illegal and that they don't agree with. I personally don't want to participate in any network where I can't control what data is stored locally.

This won't happen anyway, because it's a classic collective choice problem. If all people put some alternative software on their WIFI router at the same time, this fully decentralized mesh web would come into existence at once in most densely populated areas in Europe and the US (all big cities at least), and almost everybody would have an immediate advantage from it. But if only a few people do it first, they'll have tremendous disadvantages from it due to freeriders, abuses, etc. So it won't happen ever.

I remember to have seen a protocol for such ad hoc mesh networks that wasn't even IP-based and could be implemented on most routers. The network can dynamically self-configure, route around failures, and nodes can go in and out of existence whenever they want. It looked pretty cool but unfortunately can't remember where I've seen it. :/

Are you thinking of cjdns?

https://en.m.wikipedia.org/wiki/Cjdns

https://github.com/cjdelisle/cjdns

For me the following points come to mind reading something like this, which I think you also partly addressed:

1) Cloud hosting --> might as well use the centralized application because this is still centralized.

2) Peering your own hardware for a direct internet connection is not accessible to most people.

3) even in a p2p context, your PC or mobile device is not under your full control and especially in the US, most broadband goes through one of a small number of service providers (Comcast for example)

So I agree with you: network architecture is a major stumbling block.

On second thought, though, that still doesn't mean we shouldn't strive to create "decentralized" applications that are accessible to most average people somehow. I believe this is possible and without a huge upfront cost (besides time). They could still gain power through traction.

And as far as hosting content for others in a p2p context, I think part of the point of a scheme like that is you should never be able to know what you're hosting. It should be opaque encrypted blocks of data, right?

> Where the change will come is when we are no longer dependent on paying for network access.

When is that going to happen? All those routers, etc. don't run on fairy dust and the people that keep them serviced and running kind of enjoy doing things like being able to afford to eat, etc.

Even if you get rid of that part, you have to provide power to the devices and pay for a place to put them. Don't expect that a local telco's going to let you just slap a box on their pole rent-free. Any halfway savvy businessman is going to say "Good of humanity? That'll be $25/mo to put your box in my shop." unless there's some larger, provable benefit to him or herself.

> When is that going to happen?

It already happens/has happened with AMPRnet, HSMM, CUWiN, Freifunk, FunkFeuer, OpenWireless, Firetide, Guifi.net, Netsukuku, Ninux, Senceive, and others.

https://en.wikipedia.org/wiki/Wireless_mesh_network

When higher levels of portable energy become safer and cheaper, then it would be more likely that a small radio could be used to transmit and receive over longer distances.

If a company like Google and Facebook were to provide free adequately high-speed internet access, there'd be less incentive for a free peer-based network wherever they provided free access. Google's free wired plan is lower speed and they didn't plan to expand it; however, I'm not sure about their plans for Wifi. Neither Facebook nor Google have stated they plan to provide access everywhere. So, free access via a peer-based network would remain an incentive over paid plans for the foreseeable future.

You mention Google which is a search machine to gazillions of websites. Surely people actually want to be able to visit those sites!

Google has been much more than "a search machine" for a long time now. I assume they were referring to Google Docs, Google+, etc.

It seems reasonable to me that indexing and search could be decentralized just as well as the documents themselves.

I also think there is a generational component to this as well. You have an age group who don't know an internet prior to this centralization grab. So there is not much in the way of a standard of comparison for them.

Right. Probably the only way newer generations who've grown up in this context will come to embrace an alternative is if it at least looks like the centralized apps (on the centralized platforms) they're used to.

So you have to create a system that's decentralized and yet has the interface of a centralized application and which is distributed on centralized, mostly monopolized platforms.

So that's certainly an undertaking... but not necessarily impossible.

I completely agree, its definitely not impossible. I think just awareness would build some momentum.

> they will only switch if something else offers a clear higher value

No, that's not how it works. Sure, perceived value is important, but that's not where it starts. For people to switch to something else they need to find out about it first, and to find out about it somebody has to promote it to them. And how do you get people to promote it? You need an incentive, you need to find some way to make it worth for people to promote the software and of course it has to have a value for users to use it too.

Torrent sites, for example, benefited from promoting torrent technology.

> For people to switch to something else they need to find out about it first, and to find out about it somebody has to promote it to them

The perceived higher value itself is the incentive that will make people tell other people about it. People like to share things they really like or dislike. It gives them something to talk about ;-)

No, not really. To tell each other they first need to find out about it too and not just find out, but also try to use it. You can benefit from this, of course, but it cannot be a force moving product until it reaches some critical mass.

I was talking more in general.

You seem to talk about building a platform to replace Facebook, Snapchat, Google and others like them ? Good luck ;-)

Snapchat made it by offering easier photo options before Facebook did. Whatsapp did the same with messaging. Maybe some startup will make it with VR before the others do. But 'more power' and 'decentralized' will simply not do.

> To most people (99% ?) Facebook, Instagram, Google etc offer everything they want.

This is the classic "If I asked people what they wanted, they would've asked for a faster horse" thinking.

There are lots of things people want that Google, Instagram, etc don't offer them. Most people just don't think they're allowed to ask. I know many people who want a better job, people who want to be able to stop living on the street, people who want to eat better, people who want political representation they believe in.

Health, career mobility, and an end to poverty, and good reprentation are maybe acceptable outcomes for specific businesses. But overall the world of business owners is hurt by those things. Sick people spend more money to treat their symptoms, people without job mobility work harder for less pay, poverty forces people to spend the same money over and over again, and corrupt officials can be controlled with bribes. These things are all great for the great core of businesses from Wal-Mart to Coca-Cola to the donut shops, strip clubs, and restaurants on your street.

The way that incentive plays out is complicated, and this is just a theory. I can't prove that there's any kind of permanent system there. But that's how it seems to me.

Decentralized systems don't have that incentive. Ethereum doesn't care how easy it is for employees to quit. Email servers don't care how sick people are.

That doesn't mean those system will necessarily be empowering... "Empowering to disempowered people" is a political aim, and if you want tools to support a political aim, they have to be designed and tested for that outcome. Unfortunately, even though the decentralized tools don't care about the cultural incentives, those of us who build tools on those decentralized platforms do, and we often recapitulate the same incentive systems that Google, Facebook, etc do.

BUT, the light in the end of this whole tunnel for me is that IF you can take "empowering disempowered" as a design goal, and IF you can find some space outside of the commercial incentives to design a tool on decentralized platforms that moves that dial, THEN that tool will actually embody those values, and because it is decentralized it will not be beholden to the standard cultural pressures, and that might actually make a difference.

To some extent this happens every day in in little corners of every product and service in the world. Employees do find ways to enact their values in the work, when they have a little bit of free energy after doing what their boss likes. In theory a decentralized toolset can replace the corporate structure that makes that employee beholden to the boss, making them beholden to algorithms of their choice instead. In theory one of those algorithms would be responsible for keeping them solvent. In theory that could free them up to spend a larger percentage of their time enacting their values. In theory that could create more economic value than doing what an (largely arbitrarily selected) supervisor thinks is right.

But that's a lot of theories.

I went to the Decentralised Web conference in June. Fantastic venue, good talks, and lots of interesting people to talk with during the breaks. I blogged about the experience http://blog.markwatson.com/2016/06/action-items-after-attend...

Until recently, I was trying to go 'all in' by favoring GNU Social over centralized social media, almost always running Linux on the laptop, etc.

I have backed off somewhat, realizing that my workflow for writing books and consulting is more efficient using OSX, and sometimes Facebook and G+ are much better at connecting with friends and better for publicizing book updates, etc. than GNU Social.

I am trying to live in a practical 'middle ground' where I can get my work done and still participate in keeping the web open and decentralized.

Two new developments that are promising: a Ruby version of GNU Social that uses the same protocols that looks much more hackable, and TBL's W3C Solid project.

Facebook and goog is build by armies of full time folks with average salaries of 150k. Competing with volunteers doesn't get you far.

Even the rise of git was because of github. A for profit centralized company.

May be the best case is to have open, decentralized protocols, with for profit companies providing hosting and simple signup.

ICQ or XMPP, never quite took off like slack. Is it because of the interface?

A couple of years ago everybody had ICQ, at least here in Germany. I wouldn't say that it "didn't take off".

Same in Australia before MSN messenger (and now facebook messenger) supplanted it. I set my phone notification to the default ICQ sound and if I get a message in a meeting I see the instant wave of nostalgia across everyone's face.

According to same logic Facebook or Google would have never become what they became and we'd be living in a world of Altavista and Yahoo!

Does anyone remember the "decentralized web" of the early 2000s? Varous P2P protocols existed that allowed everyone to share content freely, and they did. Content that was not even originally in digital form was digitised and nothing but a search away. It was amazing. The copyright/media industry didn't like that. Security paranoia (possibly assisted by the industry) also heightened its demise.

I really do hope we see another "rise of P2P", but there seems to be strong commercial interests against it. (Bitcoin itself is rather commercial in nature, as it deals specifically with currency.)

There were downsides too. Getting memes from eMule was great, but imgur is so much easier.

Are you of the opinion that p2p piracy has somehow gone away?

I do have to question the notion that free entertainment is what empowers people, but I suppose it isn't the stupidest attempt at justifying piracy I've ever read.

Are you of the opinion that p2p piracy has somehow gone away?

It hasn't, but it's far diminished than what it used to be.

I do have to question the notion that free entertainment is what empowers people, but I suppose it isn't the stupidest attempt at justifying piracy I've ever read.

Not only entertainment, but information in general. Various books and journals, some of which you probably wouldn't even be able to buy a physical copy of, obscure software, etc. "Knowledge is power." Piracy isn't just movies and music...

> Piracy isn't just movies and music

You're right, of course, but this is one of those situations where 99% of cases ruined it for the rest.

And I would argue heavily that p2p is diminished in any way. I don't have anything to do with the piracy subculture and it took me under 5 minutes to find torrents of current movies, tv shows, and albums.

I remember one of the first VOIP clients (if not the first) PowWow. https://en.wikipedia.org/wiki/PowWow_(chat_program)

I remember having to install a gazillion chat clients, each with it's own peculiarities and it got fucking annoying. There were redundant connections galore with one person, but no guarantee that they would be available on all the services all the time, which turned into a chat whack-a-mole.

Decentralization sounds great, but it can be a pain in the ass, sometimes. And if you've somehow managed to set it up to feel like it's centralized then 1) it probably is and 2) you've made it easier for someone to create a one-stop-shop which is centralized, built on top of your decentralization.

Decentralized services have not been able to compete with their walled garden counterparts due to lack of resources and access to capital, as well as the coordination costs of federation. It's a structural and not a technical problem. I can't see how this will ever change.

Most decentralized services are open source projects maintained by volunteer developers. They are competing with centralized commercial projects with deep pockets and the ability to hire not only smart developers but also artists, testers, sysadmins, designers, marketers, researchers, and project managers - some of the things you need to deliver a best-in-class service to users. The odds are stacked against the decentralized service from the beginning.

Even when these projects attract commercial interest, such as Dat or Ethereum named in the article, it is not clear how their funding will be sustainable given that decentralized platforms are more difficult to monetize than centralized ones. And it's really hard to see Github as a "posterchild" for the decentralized web, since it is really a centralized service.

It has to use the existing gardens as scaffolding: embrace , extend, extinguish. Your new service has to interoperate/sync with Facebook (using screenscraping or API) in order to access your friends and feed, and then it's up to the users to choose to migrate, edge by edge, to the alternate network.

Yes on incremental migration, e.g. the IndieWeb community targets content creators, allowing them to syndicate from decentralized to centralized, so that the content system of record and ownership is under the creator's control.

I don't think so. Decentralized services never took off very much because of tremendous technical difficulties with reliable NAT traversal/hole punching. Companies like Skype spent unbelievable efforts to get around these problems and kept them a closely guarded secret. There are a few more libraries now, but it's still surprisingly difficult to achieve reliable p2p connectivity without some external, centralized server to get the process started.

This in combination with a few other problems with ISPs, slow upload speeds and the "intellectual property" watchdogs prevented 1-click installation. Without extremely easy and fast setup that even the dumbest person can manage, no software will achieve wide popularity. Shareware authors have already realized this in the 90s and this hasn't changed since then.

So my verdict is: technical reasons, mostly, although I wouldn't exclude the possibility that there were some "dark forces" behind them in some countries. Genuine P2P has always scared the shit out of traditional companies and politicians.

This isn't entirely true these days - ICE (RFC5245) has been out for 6 years and defines a pretty robust, open and tractable solution for firewall traversal. There are a bunch of libraries and apps out there that use it, including one built into most web browsers as part of WebRTC. If anything the exciting situation we're in currently is because the tech has caught up at last. Firewall traversal is no longer a black art; you just open a WebRTC data connection.

> my verdict is: technical reasons, mostly

This conclusion is not really supported by the facts.

The failure of decentralized VOIP services to achieve mass adoption was not due to technical reasons. Let us remember that SIP was standardized in RFC 2543 in 1999 and had a head start over Skype. The first STUN (Session Traversal Utilities for NAT) RFC 3489 to help solve the NAT traversal problem was published in 2003, the same year as the first release of Skype. There was no technical reason stopping the SIP community from building reliable VOIP clients to compete with Skype. One might object that due to NAT, SIP required most users to use a STUN server. But this is not a technical problem: it is similar to the situation with XMPP, which required users to register and then connect to an XMPP server, yet XMPP is still a decentralized protocol.

Besides, decentralized services encompass a lot more than just VOIP. For example, they includes instant messaging, file storage, and social networking. The first Jabber/XMPP protocol RFC was published in 2004, even though the server software had been open sourced since 1999 or so. The decentralized file storage service Tahoe-LAFS was first released in 2007. The micro-blogging service and Twitter-alternative GNU Social was launched in 2008. The Diaspora decentralized Facebook-like social network was launched in 2010. None of these services were able to compete with their commercial counterparts, yet there was nothing technical stopping them from doing so.

These services and their spiritual descendants have failed to gain mass adoption because they didn't have the same resources as their centralized commercial counterparts, and so could not offer users a better or comparable product in terms of quality, features, usability, reliability, or even "cool factor".

There is a parallel here with the mythical Year of the Linux Desktop, which never seems to arrive.

> This conclusion is not really supported by the facts.

Oh yes, it is. I have to wholeheartedly disagree with almost everything you've said, I'm afraid.

STUN requires a centralized outside server. As I said, by now there are better libraries that didn't exist ten years ago, but even they fail often. NATs behave very differently form implementation to implementation, some don't even get UPnP right, and it takes a lot of resources to get p2p reliable without special user interaction (e.g. with their home router) and without any server outside. AFAIK, there not no single library or method that works reliably without a server outside of the NAT. Also don't forget that I was talking about why decentralized p2p didn't take off, so we're talking about a of 1996-2010 not about now.

Adoption of technology does not work the way you describe, it works similarly to language change. It doesn't have much to do with resources of companies. Here is how it works in reality:

1.) If there is already an easy to use service that covers the functionality, new ones will likely fail. (Just like possible word compositions are often blocked when there is already a word for sth.)

2.) If end users cannot get it running within a few seconds, the technology will likely fail. (If it is intended for a mass market, I'm not talking about specialized tools or b2b software, of course.)

Decentralized p2p failed because of 1.) and 2.), and the reason for 2.) was primarily the lack of a cross-platform, reliable way to punch holes through NATs.

All of the examples you give fall under 1.) or 2.) respectively. On the other hand, Bitcoin is a typical example of p2p that satisfies both conditions, and that's why it was successful.

But there are are more conditions (though not the ones you suggest). One major factor was also that for an ISP, it's most desirable not to see strange server traffic from their ordinary, non-business end-consumers. So even if you succeed with NAT and complicated network topology issues, ISPs might still mess with the traffic or even block it. In fact, some infamous ISPs even disallowed running servers at any non-standard ports for some time, and used other methods to limit the Internet to passive web consumation for their customers. My ISP in the late 90s, for example, used PPP over ADSL for no technical reason, and switched off the connection once a day for no technical reason other than making it a bit harder to get a continuous server to work. And let's also not forget that most consumers nowadays are behind ADSL, where upload speed is usually limited 10 times less than download speed. That alone makes p2p harder.

Again, all mostly technical reasons. By the way, the Linux Desktop is also not successful for mostly technical reasons, contrary to what you claim. First, there is the application barrier maintained by commercial companies - a technical issue that could easily be overcome if besaid companies would work on interoperability standards (they do the opposite, of course). Second, there are numerous issues with usability on Linux that will never be resolved, because they would require everyone in the Linux world to pull on the same string at the same time. But these issues are still technical in the sense that there would be technical solutions to them, if there was some sort of "benevolent Linux dictator" who could force every programmer to use a given API like Apple forces their developers.

Because Techcrunch didn't link to any of the projects they're talking about, here they are:

https://ipfs.io/

http://dat-data.com/

https://blockstack.org/

https://www.ethereum.org/

I've been playing a bit with ZeroNet [https://zeronet.io/] lately; it's quite usable despite being a fairly new project.

Also after zeronet, don't forget about:

http://matrix.org

http://gun.js.org

http://webtorrent.io

As well as https://maidsafe.net/ and https://freenetproject.org/

I believe the internet is naturally drifting to "governed countries" (e.g. Google or Facebook) and one of the biggest drivers is not just convenience, but decreasing security of the web in general. Just like the wild West eventually became a number of states with borders, police and government. It became especially relevant in the recent years, when harvesting exploits en masse has become a huge industry. Yes, centralized "web governments" have exponentially increasing attractiveness for attackers although they can benefit from scale, applying good security practices more consistently than multiple decentralized self-governed "nodes". The criminals tend to self-organize and centralize. Why? because it's more effective. As long as there are centralized "bad guys" "good citizens" don't have any other choice than unite and develop a centralized policed "country". And "the bad guys" won't go anytime soon. The decentralized web is like youth -- it's a wonderful idealistic time but it's gone and will never return back.

Very cynical, but I tend to agree. I think the idea of the libertarian individualistic decentralized web is a non-starter for the general public. If you see the decentralized web as a matter of degree of centralization of individual services, however, a continuum where you can have a web for your family, a web for your job, your own personal web, a web run by your neighborhood, a web for your government etc. and conventions/interfaces to allow strictly regulated communication between those levels, I think you see it as an inherent quality of the web.

The problem isn't that Facebook mediates the relationship between you and the people you went to high school with, or that twitter mediates the relationship between you and Washington Post reporters; it's that once those relationships have been well-established and don't need any more protection (or need protection from Facebook itself), there's nowhere to take them to on the web. It shouldn't be easier to send baby pictures to my mother and aunts through Facebook. If one of us wants to make the effort to get some off the shelf appliance (or appliance that needs Debian installed and apt-get run periodically) and hook it to the web, and establish a hub for the family that provides most of what facebook provides, it would be better if it were both possible and easy to create and access.

The most advanced decentralized system Ive seen in web 3.0 is also the one Tim Berners-Lee talks about, solid. While still very new, I think it has more than sufficient power to take back your data from the large monopolies. One nice feature, as you'd expect from the inventor of the web, is that it's 100% backwards compatible with existing Web technology.

https://github.com/solid/solid

Link to github repo above

Matrix.org is also focused in the same domain (and could be considered more mature in some respects :) (disclaimer: i work on it.)

Have you seen our platform? We spent 5 years building it and it's also 100% backwards compatible with the Web. See my comment elsewhere in this thread about the Qbix Platform.

If anyone here knows Tim Berners-Lee and can introduce us, please hit me up, that would be extremely helpful.

I hadnt, but thank you for the pointer. Please feel free to pop into the solid gitter channel. Tim is a participant when he has time.

https://gitter.im/solid/chat

why on earth are you using a centralised chat product for Solid?! Please at least give https://riot.im a go - you can bridge it straight into Gitter for people who like the gitter UI :)

>why on earth are you using a decentralised chat product for Solid?!

Assuming you meant centralised here? ;)

oops... decentralisation on the brain; fixed!

On a side note, Riot is a site with really annoying animation.

Whatever else it may be, the animation banner just killed it for me.

In the same vein as this discussion, you can actually set up your own client - and tweak it to suit your needs, such as removing any banners, adjusting style, etc. - and still interact with others via matrix.org protocol.

It was just the website, not their software. The website put me off going any further.

What is the difference/relationship between this and https://www.w3.org/TR/activitypub/?

Solid is a larger framework of which Activity Streams can be one component. Developers are free to choose the messaging format they prefer.

It would really be interesting to see something like a "personal SOLID server" that runs on your phone, and is reachable in IPv6 only (or through tunneling via "DNS for people", replacing phonebooks conceptually and handled as a separate distributed service).

The biggest obstacle to running a personal server is that it must always be on, and always be connected. Using cell phones (which certainly have the capacity for most individuals) would make that easy, and also refocuses "my data" into a physical concept. When you used to unplug your phone, the entry in the phonebook would fail. Now, when you turn off your phone, you "go dark" in the truest sense. It would be very easy to have a secondary battery that powers a low power coprocessor optimized for this task so that even when your phone dies, the server lasts a while longer (say 48 hours?).

I think ultimately if we're going to live in a world of personal data ownership in the truest sense, our data must be something tangible we can carry with us (and that isn't an extra thing to carry).

Of course, you'd want to back up that server config and your data, so there would be monetized services for that. Celebrities and business owners would have substantially more traffic or need to be online all the time, and as a result would need to actually pay for hosting and maintenance as a function of their utility. This introduces yet another market.

And throughout this whole system, you are now explicitly trusting your data as an object to a company for their services. You are not participating on their platform while they farm your interactions.

Any thoughts or criticisms?

exactly what I had in mind, as most of user's 'smart' phone capability are under-utilized, so it's not a problem of hardware / software / usability, it's more to make it appealing for user to adopt, which is the hard problem since forever :p

"A decentralized web would be very slightly inconvenient to the people so forget about your freedom and privacy who cares about that."

More realistic title. It would make for a shorter article though.

Switching from Facebook to, say, Diaspora or something is more than just "slightly inconvenient". One of them has all of my friends on it and the other has none.

Two ideas for how we can help decentralization win.

1. Don't just clone an existing app.

The two decentralized protocols with the most adoption by far are Bitcoin and BitTorrent. Both of them have capabilities that no centralized system can match. Same with Ethereum, Namecoin, etc. So the goal of a decentralized system should be to exceed the functionality of what came before, not just to clone an existing centralized service.

2. Run your open source project like a startup.

That means gathering metrics, tracking week-on-week growth, getting press hits, creating a brand, caring about product and UX.

I've seen cool decentralized projects get no adoption because the people running them focus on the technical aspects and ignore things like usability. BitMessage, for example, is impressive technology, but the client is called "PyBitMessage" and it looks lolbad: http://i.imgur.com/ydY6PIX.png

I'm totally with you on number 2 - but don't those things require money? UX, design, consistent look, all require money. And honestly, the press is a lot more likely to talk about you if you have money invested and shiny offices. To get such things, you have to have a profoundly interesting argument to investors, which usually does not rhyme with "we want to free the people, privacy, etc". There are, of course, exceptions.

Coding costs money to.

You have to motivate designers etc. in a similar way as you motivate coders to work on the project.

Of course, and I propose this cautiously, because you have to be careful who we're talking about, you could support/encourage startups which ARE profit-motivated businesses to use models which develop open, decentralized software.

GitLab and Sandstorm.io are both examples of startups, which make open sourcing their core 'product' a part of the plan.

> but don't those things require money?

Congrats, you've found the reason why a 100% decentralized web is more like fairy tale. The reason why vast majority of people use services like Facebook, SnapChat or Uber is because they are so easy to use. Good UX is not free, service stability is not free and new features are also not free.

People don't care about the underlying technology. If one wants a decentralized web, they must demonstrate a 10x increase, not a tiny bit shitty alternative service.

I would argue that the most adopted decrentralized protocol is probably e-mail. Well, it's more of a federated network rather than purely decentralized, but the article does not really make a strong disctinction here.

>The two decentralized protocols with the most adoption by far are Bitcoin and BitTorrent.

What about email? It's become quite centralised in practice, but it is ultimately a decentralised protocol, and it is used by pretty much everyone on the internet.

And even in its semi-centralised form, it's nowhere near the walled garden of Facebook.

> What about email?

That's a really good idea! Email is an good existing source of messaging and "identity." People can choose to participate in the "network" either using one of the big hosted providers, a privacy-oriented email host, and eventually over time, using self-hosted email.

Project idea: what if all the social shit could be sent over email? You'd need to setup your email client to hide emails that represent the "data packets" for the social communications so they don't pollute your INBOX. A completely separate personal browsing interface (client side .js) would be used to browse the social stream you receive at your email address.

It's already happening a bit like that with the "Social" tab in gmail, but instead of containing links to walled garden URLs, the proposed new "social email" packets will contain the full content and won't require access to any external servers. Perhaps we wouldn't even need a "central" browsing interface, since each social item can come with its own .js to render a simple UI adapted for the social item (e.g. message, photos, album, event, links).

To make the picture more concrete, let's analyze the Event data type. Alice wants to host a party, so she invites Bob, Betty, and Boris, by sending each of them a paragraph of invitation text, some images(<1MB), two RSVP links I'm going || Not going, and a [FWD] button. When Bob clicks "I'm going" Alice is notified by email. When Bob clicks [FWD] and sends to Charlie, and Charlie clicks "I'm going" Alice will be notified again. Social functionality. No central servers.

> Alice wants to host a party, so she invites Bob, Betty, and Boris, by sending each of them a paragraph of invitation...

Yeah, contrast it with reality where: Alice wants to host a party, so she invites Bob, Betty, and Boris, by sending each of them invitation via Facebook.

Or: Alice wants to host a party, so she invites Bob, Betty, and Boris, by sending them a smart contract between all of them containing 3 Bitcoin and she will only sign if they show up. And then they all split the amount in the address.

It is behaviourally impossible to switch to Diaspora, when all your friends and family are on Facebook.

Since 0.6.0 which came out in May, the UI of PyBitmessage has been redesigned.

And if/when you reach this decentralized utopia, it will all be undermined by the browsers & clients that have evolved into spyware collecting every action you take.

The problem with this plan is that it doesn't really tackle the first rule of the Internet: spam and abuse make everything suck.

The last really popular decentralized service was email, and users migrated to large providers because they had better spam filters. Also, setting up your own mail server has become increasingly difficult, due in part to anti-spam measures. As a side effect, email has become pretty lossy.

Centralized services are far from perfect on this, but they have a somewhat easier time of it. They can fund dedicated teams to deal with abuse. So, the result is feudalism, where you join a larger organization that provides some measure of protection. (Not enough protection, according to many people.)

I'm hopeful that Sandstorm (or something like it) will make it easier for people to run their own server-side software, but it seems more for private use; you can publish a blog but it's not designed for large-scale sharing like a social network. That still seems like an unsolved problem.

There are many other players in this space - yunohost.org, cloudron.io etc. https://github.com/Kickball/awesome-selfhosted has a better list

I don't think that's the problem with these plans.

Imagine that email didn't exist, and you were designing it from scratch. There are many easy solutions to the spam problem if you start with a clean slate. The problem is that nobody would ever adopt email today if it was new.

More recent decentralized solutions haven't failed because of spam, but through lack of adoption.

Ironically, GitHub is a prime example of why centralisation is harmful. If GitHub suddenly disappeared tomorrow (or even just went offline for a few hours), many software build processes in the world would grind to a halt. Sure, because Git is a decentralised protocol eventually it'd all be brought back up again but it'd cause a major disruption and at the very least create enormous costs.

You could easily move tomorrow to another git based service. The only stuff missing would be issue management.

And your wiki, and all links leading to your project and all tooling you've built around.

No idea about the other things, but the wiki is just an additional git repo that you can check out like any other one.

And then you still need the infrastructure to run it on a site where contributors already have an account or don't mind the extra burden of registration.

People moved away from sf fast enough im pretty sure the same thing would occur if things turn sour for github.

Many rely on Github for more than source control hosting and issues. It's really not a non-issue.

I would like to have some basic issue tracking in git like in fossil (www.fossil-scm.org if you don't know, by the creator of sqlite). It's a shame that git has eclipsed all other options.

You can use distributed bug trackers with git, like https://github.com/tst2005/nitpick or http://www.bugseverywhere.org

Care to elaborate ?

There are a lot of projects that use github-pages for hosting the main site and documentation, as well as fairly tight links with continuous integration/release pipelines.

I don't necessarily think this is a bad thing, but it is a valid concern.

Except there are a lot of OSS and side projects hosted on github. It will take a long time for these to find their backups and move to another service. And some will lose source code due to faulty or outdated backups.

When you do a git clone you should always have a copy of the whole git on your dev computer so I am not sure why they would have to look for their backups.

A an enterprise Github subscriber, we would lose user authentication and authorization.

Fair enough but dont other github alternatives provide that?

Or if GitHub became Sourceforge.

Didn't GitHub jump the shark already after that notorious faux diversity presentation?

Not really, SourceForge bundled adware with downloads which effected all end users, GitHub did some daft presentation that had zero impact on anyone who didn't see it and not much on those of us who did.

Did it have zero impact? I wonder if white men at GitHub have the same promotion opportunities; I wonder if they face subtle forms of discrimination in the workplace.

I am certain that some white men don't even bother applying there, because they don't wish to handle a prejudicial environment.

I would disagree with that. Github shows a lot of the benefits of centralization, where they get network effects from forks and pull requests when these do not have to cross git server boundaries.

Using external services in your build process has little to do with centralization. People that do not realize that having an external dependency in there is bad, will also not realize that having two or more of them is bad. That does not mean that you are not allowed to sync stuff in from github, but your release should be buildable without internet connection from local data.

The real issue is that we need to make it illegal to give out free content, unless it's distributed under an open license. Otherwise advertising-supported businesses will always be able to outcompete everyone else via dumping, and these businesses always benefit from being increasingly centralized. That's why we don't have liberal newspapers anymore in the U.S., because ad supported papers put them all out of business.

At the very least people need to be trained that it's unethical to surf the web without using an adblocker.

The centralization is a good point, but I wanted to ask about why

> we don't have liberal newspapers anymore in the U.S.

Most conservatives would say the papers and media outlets are _all_ liberal. Would you expand on that spectrum?

> it's unethical to surf the web without using an adblocker

It's also unethical to insert malware into ads, or to hire syndication services that allow it. I'll continue to block until that's fixed.

>> we don't have liberal newspapers anymore in the U.S. >Most conservatives would say the papers and media outlets are _all_ liberal.

I think this effect can be best demonstrated by a conversation I had with a Google fan last week. In particular, I've had an ongoing banter with him about his choice to refer to The Verge as "iVerge". I find this very odd, that people believe The Verge has a strong Apple bias.

You see, The Verge regularly gets exclusive interviews and opportunities from Google. When the Pixel was announced, The Verge staff had already been sent a demo unit, and they had a ton of quotes from senior Google employees available to use in their article, written in advance. Why would an Apple-biased blog be given so much direct access to Google? And this isn't new by any stretch of the imagination.

What I've come to recognize is that The Verge will shill for anyone who gives them the goods. Exclusives are as valuable as Google or Apple outright cutting them a check, because it guarantees a ton of page views. Apple and Google both are pretty manipulative when it comes to their press, and unsurprisingly, The Verge rarely posts hard-hitting critical material of either, because hurting a company's reputation in an article is a great way to get blacklisted by their PR team.

Anyways, the point of the story: The Google fan stated that even though The Verge seemed equally positive about both companies, that it was Google, in his opinion, which was innovative and did amazing things. So by giving Apple similar coverage, The Verge was being biased towards Apple. Because in his opinion, Apple didn't deserve the praise, but Google did.

If you are a biased viewer, a moderate voice can be quickly confused with bias for the other side. This is also, for instance, why liberals think I'm a conservative, and conservatives think I'm a liberal.

This brings up the larger view of the problem - exclusives.

If someone gets an exclusive review before the actual launch and doesn't even have the commodity to spend enough time with the product, that will absolutely have an effect to the review result.

The good old cool headed reviewing style like Anandtech, where they take the time, perhaps even wait for an software update or two, then state the verdict is more reliable. Or perhaps even some reviewers where they repeat the test after 6 months and review the unit how it stood the test of time and what's changed. That's proper, adult way to examine something.

But it sure as hell doesn't bring up all the marketing hype and SEO numbers. Modern media is SHIT.

For the reason above, I'd say we should learn to treat an exclusively granted review or interview to be a sponsored post. Because an exclusive interview with Sundar Pichai is not substantially distinguishable from a paycheck in the long run for an ad supported blog.

> we should learn to treat an exclusively granted review or interview to be a sponsored post.

I agree, although unless the authors will state this in the headline, there's still the element of dishonesty left.

Or we start teaching the difference between ads/sponsored articles and true ones in schools.

> Most conservatives would say the papers and media outlets are _all_ liberal. Would you expand on that spectrum?

Really? Isn't Fox ranked at like number 2? If anyone think that's liberal they need to realign their gauges

It may be worth noting that the OP referred solely to newspapers, and Fox is a TV channel. Noting that the parent did say "papers and media outlets", which is definitely less true, because of Fox.

Note that Fox is likely so popular because of how few strongly conservative-biased media options there are.

Looks like #1 for cable news right now, but compare e.g. http://www.adweek.com/tvnewser/scoreboard-wednesday-october-... and http://www.adweek.com/tvnewser/evening-news-ratings-week-of-... and you'll see that broadcast news totally crushes cable news.

> Most conservatives would say the papers and media outlets are _all_ liberal. Would you expand on that spectrum?

How many active communist newspapers are there left in the U.S.? As far as I know zero, but I could be mistaken.

And yet those are what gave us our labor laws, integrated workplaces, women's rights, etc.

Newspapers have always been ad supported. They charged for the paper, yes, but they always depended on ad revenue to survive.

In fact ads, job ads and classified ads had always (well at least last 50 years) the main revenue source. Sale prices are often a smaller, but still relevant, factor.

The newspaper crisis actually is caused by job and classified ads going away.

Actually, not so much. There was some but largely limited advertising in the 19th century, but the start date was February of 1860, when The Atlantic accepted its first print ad. The practice exploded in the late 19th and early 20th century, with the rise of the factory process and widespread distribution networks, such that goods could be centrally produced and sold in distant markets.

Alexander Holt discussed this in a 1909 lecture, "Commercialism and Journalism", presented at the University of California and published as a small book. It's available on the Internet Archive, and the first section has a considerable overview of advertising practices and statistics:

https://archive.org/stream/commercialismjou00holtuoft#page/n...

Quoting:

In olden times the dailies carried only a very little advertising --- a few legal notices, an appeal for the return of a strayed cow, or a house for sale. It is only within the past fifty years that advertising as a means of bringing together the producer and consumer began. And, curiously enough, the men who first began to appreciate the immense selling-power that lay in the printed advertisment were "makers", or "fakirs", of patent medicines. The beginning of modern advertising is in fact synchronous with the beginnings of the patent-medicine business.

The remainder of the essay is mostly a cautionary exploration of the consequences of commercially-financed press.

>, but the start date was February of 1860, when The Atlantic accepted its first print ad

Can you explain the criteria for "advertising" such that 1860 is marked as the significant date? There were paid ads in newspapers/pamphlets in the 1600s:

https://www.brent.gov.uk/media/387509/Newspaper_advertising_...

That said, I think it's reasonable to set boundary date of when newspapers switched the majority of their sustainable revenue from subscriptions to ads. However, I'm not sure The Atlantic in 1860 is an example of that.

Wasn't political party support also a source of income? In those more honest days, papers didn't pretend they were "objective", and you'll see traces of this today in some papers that retain "Republican" or "Democrat" in their names (Whigs, not that I remember. :-)

Very much so. I was going to mention that but figured I'd rambled on enough already.

You might enjoy a piece I wrote about a year ago on J.P. Morgan and the purported "Banker's Manifesto" of 1892. (It proves to be a hoax.) The investigation took me through quite a few old newspaper archives, many of them political organs.

https://www.reddit.com/r/dredmorbius/comments/39w8u4/jp_morg...

The Arizona Republic, commenting on its own endorsement of Hillary Clinton, a Democrat, the first presidental endorsement for that party in the paper's 150+ year history, noted that it was originally known as the Arizona RepublicAN.

Consider what it takes to have a substantial advertising market: you need goods, you need buyers, you need sellers who are providing a great deal of a good (or are selling very high-ticket items for which buyers are thin), and you need means for transporting both the advertisments and the goods to buyers.

Hamilton Holt, a publisher himself (of The Independent magazine) starts off the linked work by detailing this. The first 10-15 pages cover a lot of ground, I highly recommend you read them if you haven't. Keep in mind that Holt is largely limiting himself to US practice, so specifics of earlier European publishers may be missed. But he is also focusing on the business volume of advertising, not simply the existence of occasional adverts or notices in publications.

What your link describes in considerable detail are early instances of advertising, and a few specific publications which carried them. They're drawn from across England and France, as well as the US, but don't give an impression of a thriving advertising industry, which is precisely what Holt is talking about (there are a slew of similar and related works at TIA from the 1890s to 1920s covering the boom, many offering advice to publishers, or more often, businessmen who are looking at this newfangled thing called "advertising" -- it's surprisingly interesting reading, at least for me).

I'd argue that this passage from your reference largely makes my point: that advertising wasn't a truly significant force until the 1850s, due to a confluence of factors. Again, it's patent medicines (small packages, complex product that has to be marketed, travels well, premium price, mass production) which launched the phenomenon:

The abolition of the advertising tax in 1853, the duty on newspapers in 1855 and the duty on paper in 1861 created a new environment for advertisers and publishers alike. Thomas Holloway (1800-1883), a purveyor of quack pills, was spending over £30,000 a year on advertising by 1855.

QED.

>substantial advertising market [...] thriving advertising industry, [...]

Whenever I see that "newspapers were always ad supported", I take it to mean the mainstream press and not the early financial papers that went to subscribers on Wall Street or the political news that was paid for by wealthy patrons. (Mainstream == readership of working class people.)

The "penny press" in the 1830s[1] is an example of lowering the price from 6 pennies to 1 which enabled a wider readership. Advertising made those economics of mainstream publishing possible 30 years before Holt's focus on an "advertising industry". I understand Holt's logic but to me, the idea of advertising quantity is not relevant to how the earlier advertising created the financial support for mainstream newspapers.

[1]https://en.wikipedia.org/wiki/Penny_press

Holt addresses this, the advertising vs. subscriptions rates dynamic and how that affects circulation, the first fully free advertising circulars, and the 30x growth of publishing 1850 - 1905. That's 3000% growth, mind, not 30%.

> The real issue is that we need to make it illegal to give out free content

The First Amendment guarantees that I can publish freely, with or without charging for access

> At the very least people need to be trained that it's unethical to surf the web without using an adblocker.

It's also unethical that ad Networks spread malware (that happened multiple times) and cause costs from my mobile company due to all those auto playing videos. It's also unethical that they track me all over the web.

In a newspaper I'm not tracked and no ads jump at me and I get to throw away all those stupid flyers without anybody judging my ethics.

Eh, you are not the one with problems noticing the without part. Pattern matching is powerful.

I lived in one totalitarian society, in Eastern Europe, which started on a good principles of general equality, but with caveat that people aren't really ready for the bright future, so we should train them to do the right thing, and make illegal to do things that leaders don't like.

Why we don't, instead, make that free, decentralized Web 3.0 better than the current one, so the people will willingly use it? To borrow the example from the article, people are using git over subversion not because it's more ethical, but because it's more convenient.

It's easy sitting on a high horse and preaching about freedom and liberty. Making something that ordinary people, who don't give a flying fuck about ethical implication of software use but just want to see the picture of the grandchild or want to share the pie recipe on the internet would want to use without being "trained" or marked as "illegal" - well, that's the hard part.

The time spent making it better isn't free.

You realize that your own post would have been deemed illegal under that rule?

We don't need more censorship, thanks. If you want to abolish advertising, then target it specifically.

Ad supported business is a side effect of business / marketing guys taking existing business model to a new medium. There's nothing wrong with it, it's the dumbest model ever to generate revenue. That's not the current web's or law system's fault that people can't come up with a better alternative and bigger newspapers are so clueless how to make money by other means.

I think the real issue is that 99.9% of internet users are not tech savvy and don't really care how the content pops up on their screen. The internet and content is now an utility, like tap water. It's a hard sell to tell everybody to drill their own wells.

How is it dumping? Dumping means artificially reducing the price, i.e. running at a loss.

Aside, this destroys free speech.

In my opinion one of the disturbing trends increasingly presenting a "usability" issue is the insistence of sites putting content behind a login wall, Pinterest, LinkedIn, Glasdoor etc. Some of these make a subset of content viewable but then prohibit me from scrolling further until I sign up. Usually the sign up requires selecting one of FB, Google, et al as an Oauth provider. So by extension you kind of have centralized control of third party content as well.

https://whispersystems.org/blog/the-ecosystem-is-moving/

> We got to the first production version of IP, and have been trying for the past 20 years to switch to a second production version of IP with limited success. We got to HTTP version 1.1 in 1997, and have been stuck there until now. Likewise, SMTP, IRC, DNS, XMPP, are all similarly frozen in time circa the late 1990s. To answer his question, that's how far the internet got. It got to the late 90s.

> That has taken us pretty far, but it's undeniable that once you federate your protocol, it becomes very difficult to make changes. And right now, at the application level, things that stand still don't fare very well in a world where the ecosystem is moving.

> Indeed, cannibalizing a federated application-layer protocol into a centralized service is almost a sure recipe for a successful consumer product today.

The is awesome! I hope they're able to bring their vision of a decentralized web to fruition!

The biggest outstanding problem that nobody seems to be talking about though is how to monetize a decentralized business. With the current web there are lots of options that have varying degrees of nastiness for their users but ultimately pay developer salaries: 3rd party ads, IP/API licensing, pay for product, data collection, in-app purchases, e-commerce, etc. Since not all of these carry over well to a decentralized web, how can it be profitable for companies?

The problem of building a generic, distributed platform (or at least some technology for decentralized services) probably comes first but putting money behind its development couldn't hurt. I think if engineers won't be able to make money with it, then it will be much harder to sustain development in the long run.

Any ideas for this part?

There's no need to monetize user generated content. That's what these companies do. They make people generate content for them for free and use it to sell ads while creating an orwellian dystopia.

I wasn't really suggesting that businesses keep using the same web 2.0 models. Just fishing for ideas for the distributed web. If teams are actually going to commit thousands or millions of work hours into making products, then there should be a way for them to be rewarded for their job. Making money isn't inherently evil, but making it at the users' expense is.

idk, there could be client-side Bitcoin mining if somehow you can find a breakthrough mining algorithm (maybe using Algebraic Cryptography or Satisfiability Theory) that runs an order of magnitude faster than current and users are OK with running it.

e-commerce could have some kind of payment system with escrow where you could charge a small operating tax. Hard to ensure everything works though since all code is running client-side, which can't be trusted.

pay per download probably still works.

others?

I agree that ads, tracking, IAP, and other stuff should be left out.

It's weird to think about services which as the creator you really have no control over once it's released (I'm thinking of Freenet or GNU social style services). There aren't any services that I know of which are both 100% distributed and making money on the regular.

Not really well thought out ideas, just putting stuff out there.

The current standard is "ICOs" (initial coin offerings).

The strategy there is typically to develop a new protocol/service that's openly shared, and selling off a currency tied to an organization responsible for spearheading it. The currencies being some mixture of credit, stock, and increasingly abstract forms of value.

Decentralized protocols still have plenty of room for centralized organizations that extract value from acting as a focal point for community and services built on top of the protocol.

A decentralised web is not viable until we can create an NP soft or better mesh network. Until then the network load to manage itself crushes it before it can walk.

decentralized routing is a problem separate from decentralized hosting on a core-shell topology network, i.e. the current internet.

It is not viable until there is an order of magnitude cheaper storage (e.g $1/1TB) and cheaper/faster bandwidth. For the latter it is becoming more expensive and slower lately.

+1 for bringing up mesh networks. I have been reading about mesh network projects in Germany and other places. What I think is needed to increase adoption is something like the raspberry Pi, a cheap and open mesh router that would allow setting up community networks, with hooks for volunteers to provide low bandwidth gateways to the web, tools for sharing locally hosted copies of Wikipedia, projects for local social media, market places for local commerce, etc.

When there is a low cost system available, then I can visualize myself starting a class at the local library, etc.

In an increasingly centralized world (central banks, the awful one worked government philosophies, winner take all hubs on the web, etc.) it is too easy to lose track of the fact that local issues, economy, and life are more important than global issues.

There are many millions of people who don't have the problems, that can be solved by mesh networks. They live in countries, where ISPs are not overregulated and are very competitive, bringing cheap high quality high speed internet to every household. In fact, those ISPs and the whole broadband market there emerged from similar communities of local networks you describe, except they weren't mesh networks, but wired networks.

And yet there is no decentralized web there either. Because it's not technology, that's stopping it from happening.

Thanks for your comment. I see mesh networks as an alternative to be used alongside the web. Also, in the USA, local mesh networks were useful in dealing with the aftermath of Hurricane Sandy, as another example of utility of mesh networks.

> tools for sharing locally hosted copies of Wikipedia,

Why?

It's much easier for the ISP to peer with Wikipedia/etc.

Why would you want a low bandwidth gateway to the web?

I don't understand the advantages.

Valid question. I was thinking about maintaining local copies of useful information for emergency use. Most things are going well in the world right now, but it is possible that there could be long term outages in infrastructure.

I think that article is missing one important thing. "Web 3.0" needs a killer app! Something which is not simple to duplicate in current centralized web and still It's very attractive to web users.

It still pains me to see that Mozilla Persona failed.

It was terrible to see that happen, but it looks like getting out of silos built by the likes of Google, Facebook, etc., is becoming more difficult with time.

If something has to be built, it has to be done by common people. There's no point looking at large corporates for these.

I see a lot of HNers lamenting the loss of persona. The code's available, so what's the major issue preventing its continued use?

Is it the use of de facto addresses, like the fallback login.persona.org? Couldn't that be mitigated by either establishing a new de facto standard (either commercial or community run), or by adding a fallback field to the protocol?

It seems to me that one significant barrier is that most people don't have a static IP. If we can move to IPv6, will ISPs start issuing static IPs for everyone by default?

Not having a static ipv4 ip is not a show stopper. Nat traversal works fine across most carrier grade nats. I agree that ipv6 helps.

Doesn't dynamic DNS solve this problem? I have used that for a service on a home server set up for friends. I haven't had any connectivity problems using the dynamic DNS address, although, to be fair my dynamic IP address has stayed remarkably static.

It does, but it's a hacky workaround and fails sooner or later unless you're using a paid service.

A static IP brings along privacy and tracking concerns when public facing servers/services are mixed with personal browsing and use. Even if ISPs are handing out IPv6 addresses, it would be better for people to get addresses that do change often or on demand.

You can't reasonably expect ISPs to provide anonymity; that would be like asking the Post Office to change your mailing address on demand.

It makes more sense for ISPs to focus on efficient delivery, while anonymity is handled by things like overlay networks.

In my experience, one of the largest hurdles decentralization faces is agent identity. Or simply put, how do you determine who is who on the network. The original Internet failed to address this problem.

The successful solutions I have seen to this issue have employed centralization. SSL certificates from Verisign and login with Google Sign In so consumers can trust your website.

Don't confuse this with distributed authentication. They exist , but I would say difficult to use.

I am talking about identity.

Does this contact come back with the same public RSA key each time? Could they sign a small piece of data with the key? It probably [is/isn't] them, then.

In Synchrony[1] this forms your address (like an email address but for editing web pages / chat / RTC teleconferencing / multi-user JS/WebAssembly applications), which contacts can then put a name to in the UI by mousing over.

The address has the form network_name/node_id/user_id, where network_name is an opt-in namespace (so Synchrony can select routing tables based on their overall trustworthiness), node_id is 160 bits based on the SHA1 of "ip:port:pubkey" which peer nodes can then verify, preventing one another from assuming node IDs that correspond to popular URLs, so no one can computationally inexpensively monitor popular URLs for financial gain[2].

Note that the RSA keypair is per-instance of the software and not per user-identity - that when we ask a node if it can sign for some data it's the installation rather than the individual user identity being checked.

[1] https://github.com/psybernetics/synchrony [2] http://www.tm.uka.de/doc/SKademlia_2007.pdf

Keypairs are not useful identities though. They don't encode any meaningful attributes (e.g. name) and it's too hard to manage them.

The nature of the net us to centralize and then decentralize, it's a cycle. See 90s AOL vs the open net as an example. At this point it's more about usability than it is about access. The open tools that are being built still have a ways to go towards fluid user on-boarding and compelling retention experiences.

When has it ever decentralized? The nature is that new decentralised systems get invented, then they get centralized. The outbursts of decentralization keep getting smaller and less frequent, as programming software gets more complex with more dependencies and requirements (security, internationalization, etc)

There's very little evidence to support this claim. Some things were decentralized in the past, now most things are centralized. That does not establish a cycle.

The Internet has many centers and choke points, most of which gather in the U.S. The Internet needs country level network diversity and more stuff happening on the edge of networks instead of the center. Maciej Ceglowski explains it better than I:

The Internet With a Human Face: https://www.youtube.com/watch?v=fWFo1VaQNmU

It's nice, how alive and kicking IRC still is. If you are into startups, there is a channel called #startups on freenode that has a lot of likeminded people.

I think the European union should fund the research and development of a decentralized web. Because right now all their information is flowing to the US.

Decentralization won't stop information flowing to the US. We'll just have to buy additional peerage points. Our budget and desire to spy on the world is greater than the world's resolve to not be spied on.

Well, initially the web was a fantastic place to be. People used the web for entertainment and people used to trust people online. Every website was unique and handcrafted.

Then, as evil started going online in the form of harassment, fraud, scams, spreading malware... people started to be reluctant of following any link, and people started to seek for "trusted websites".

In addition, people gave away the ability of customizing webpages in return for searching people by name, real time cross platform publishing, and photo sharing with access control lists.

Also, interoperaibilty. The major problem of all these major services is not just centralization, but the fact that they can't even communicate with each other. Their stupid walled garden nature and the fact that they on purpose avoid open standards prevents it. That's what disgusts me the most about Whatsapp, Hangouts, Skype and etc. They are stuck in the non interoperable stone age of computing, while e-mail, much older technology managed to break through years ago as federated and interoperable standard.

The internet is decentralized. The thing is that you need to get the data from person A to person B. And neither A nor B can alone afford the infrastructure to enable that.

Many people think Facebook is the web, just like people think the web is the Internet. I'm sure both web and Internet will outlive Facebook though. Facebook have introduced a lot of new people to the web/Internet. It's just a matter of time though, until those people discover that there's more, like Youtube for example.

All this has happened before, and all of it will happen again.

Decentralizing will be great until people realize that they need access to information outside their enclaves, then there will be another push for centralization.

Additionally, who are these People the author writes about? All of the tools cited, while freely available to anyone, are geared toward power users in installation and operation. That doesn't sound like "The People" to me; it sounds more like an elite group (although a pretty large and diverse elite group, within itself).

If all you're going to do is stand by the highway and wave signs and shout slogans, I wish you well with that plan but I don't think it's going to work. The people you're opposing (who are actually people, too) have got better and more convincing arguments, as well as positive results that matter to their customers.

This remains a bit of a silly argument. It is like saying we can wipe out poverty if only more people were able to get rich: the way that people progress is through collective provision.

That means the state, or associations of people who value privacy (who may not all be technically savvy!) who can pool their resources and offer something normal people can use. This, unlike the decentralised web, could engender the same kind of network effect that whatsapp and facebook benefit from.

I've written about this and heard more people make the same argument lately: it isn't technology that will bring about privacy on the internet, but a democratic organisation, founded by concerned citizens, governed by anyone who has an interest in promoting privacy and run on behalf of anyone who wants to benefit from it.

There are a lot of benefits to this approach: you lose the engineering challenges of VC backed companies (no data mining operation) and of cyberpunks setting out to let everyone be their own little island (protocols and amateur operators). Jurisdictions can offer different advantages, like company forms with regulated purposes written into their founding documents, or more favourable privacy laws.

Off the shelf software can scale all the way that is required for this sort of thing (think apache kafka as the basis for a messaging system like WhatsApp) and the organisation would only have to be financially sustainable like Mozilla, rather than profitable like Facebook.

It also puts privacy into the hands of normal people, who can take part in governance and protect their data that way, rather than having to learn techie things they don't care about.

Talking about the decentralised web is great, but we often forget what motivates people to want to decentralise in the first place: the fact that their desires are not something the market can cater for, and that what they believe should be their right is simply not available any other way. If we want privacy, we should think about all the means by which we can provide it in the general case, including by collectively owning the means of communication.

Where does the world wide web fit into your model of the world? (it didn't need "collective provision")

All the article proposes is that we don't have our data locked up, that it's not centralized, so that we have control.

This is a pretty simply concept I think you're the only one here who is confused. Your comparison to poverty doesn't make sense, privacy isn't constrained by resources.

Privacy is constrained by technical skills in most people's model. Thats where the scarcity lies.

People talk about it as a software problem, but don't understand that lots of people have not even a slight grasp of why Facebook and the other silos are bad.

The web fits the analogy perfectly well: it has many actors and many near monopolies. This is not unprecedented: just as most people in the 19th century had houses and access to roads, people have computers and the internet now. Neither precludes domination by large companies or the very wealthy, but domination can be precludes by collective endeavours. Landlords were thwarted by building societies and mutuals, coercive employers thwarted by cooperatives and unions. Similarly, platform cooperatives can solve the problem of people's private information being commodified. Lots of software approaches have been tried to solve this problem but none have succeeded, even with sponsorship from large companies (think xmpp and google).

As yet, nobody has tried my approach because it isn't a natural avenue for exploration among people whose sole consideration is technical implementation. It has a high initial barrier because it demands a multidisciplinary approach, but once established it would only require a small team (think WhatsApp and its dozen or so employees).

We should not be worried about chat apps, but rather the cancer that is called Facebook

The example of GitHub they use is actually telling. Git is decentralized, so why use GitHub? The answer may be surprising:

It's because social has not been decentralized.

Yes, status.net existed for a while but it was just about publishing short updates. Also, a few years ago several guys from my college raised money to make Diaspora. That didn't fulfill the promise of a decentralized social web either.

Bitcoin is decentralized money. Email is decentralized communication. The Web is decentralized publishing. But what about decentralized social?

Ideally, it should start with the Web and enhance it. It should:

+ Work on every device out of the box, and take advantage of the special characteristics of each device (eg mobile phones are private to the user, and work as endpoints for text message invitations with auto-confirming links https://qbix.com/platform/features/invitations )

+ Be as easy to install as WordPress by organizations, who can choose the hosting provider to grow their community, and be able to move it anytime

+ Seamlessly support user identity AND contact lists across domains. Be able to sync with personal address books, social network friend lists, etc.(https://qbix.com/platform/features)

+ Seamlessly support a standardized access control model, ideally where roles correspond to contact groups or friend lists. (https://qbix.com/platform/features/contacts)

+ Be modular, so developers working for organizations can easily use components, developed and maintained by different developers, in the organization's social apps.

+ Allow people to subscribe to certain streams of information. Take care of real time updates via WebSockets for online users while delivering offline notifications for people who subscribed, via text, email or native notifications. (https://qbix.com/platform/features/streams)

+ If done correctly, such a system would decentralize search engines and social networking sites, allowing local communities to get massive value from being networked without having to send all their signals to California and back, or connect to Facebook's "web for India" or "web for Africa" just to organnize meetings or talk to someone next door. (https://qbix.com/platform/features/distributed)

That is what we built. And we are slowly rolling it out. And it's free and open source. You can downoad it right now and play with it:

http://qbix.com/platform

(It took us about 5 years to make it, so some screenshots are a bit dated.)

If anyone here knows Tim Berners-Lee and can introduce us, that would be extremely helpful.

Don't forget that for email to get popular, it had to be mostly centralized (Hotmail, Gmail, etc), we are lucky it's still at least federated.

Bitcoin, same story -- mass adoption turns out to be dependent on service providers hosting the wallet, and most users are not miners, eroding many of the decentralization benefits.

I mean decentralized in the same way the Web and Email are decentralized. Not all the way to the end-user nodes, but to the servers. Any number of people can bring up as many new servers as they want, and they control the server and what's on it. In our Platform that corresponds to Communities. An end-user can have accounts in an unlimited number of communities and seamlessly use their identity and contacts across communities and widgets, even on the same page.

For example a dentist can include a Maps widget and the user can get directions while the maps widget knows the user's address and the dentist knows his medical history and both can display it to him on the same page instantly without either one knowing the other's info. It does this using iframes.

I also mean enabling decentralized networking topologies, eg collaborating on a document without the signal having to go to Google Docs, or planning get-togethers and coordinating efforts in your African village via running various social apps on the the local network.

That's the type of decentralization that is actually useful.

There's still a lot of value in a mostly centralized network operating on a protocol designed for a decentralized one. That means that people who want to opt out of centralization can, and it limits the walled garden effect.

Don't forget that for email to get popular, it had to be mostly centralized (Hotmail, Gmail, etc)...

I'm not so sure. Before Hotmail and Gmail, every ISP provided email accounts to its customers. Desktop POP3 clients were the norm, and email was as popular as the Internet was. Centralization wasn't needed for email popularity.

Hotmail, Gmail etc. have now taken over, perhaps partly due to the economy of scale in dealing with the spam problem. But the smaller players are still present and working, so I'm not sure that email would fail without today's major providers.

And most companies still maintain their own mail servers, I'd say the majority of email is through decentralized servers.

In addition to spam/abuse and the difficulty of setting up a server, another area of centralization that hasn't yet been solved is payments. If we want a truly decentralized web, we need a way for computers to posses and spend money without human intervention. This compounds the difficult of spam/abuse and server setup, but is still a huge factor in why the web is the way it is today.

What kinds of business models will decentralized applications have? Any good examples?

A decentralized web seems like a logical next-step. Although, it's hard for it to happen just by developer power alone. Once viable business models and success stories emerge around decentralized applications (like ads on P2P file sharing client?), IMO, this will really pick up steam.

So far, nobody has a federated social network that 1) doesn't suck, and 2) has a worthwhile user base.

Someone in college should take one of the federated systems, polish it up, and market it to fraternities and sororities at name schools as FratNet. This would give all the cool kids a private social network that they controlled.

Networks don't tend to stay siloed.

Very quickly members move on to graduate schools and then jobs. The social network boundaries expand beyond their original fraternities and sororities. You end up with one large social network.

That's the point. As Zuckerberg taught us, you have to onboard the cool kids first. Then you grow. The federated network people don't get this.

One area that may be ripe for the picking is video chat, as this article starts out describing. I think everyone is sick of having half a different applications to talk to their friend. Why can't this be a protocol like email?

I think the technologies exist and have existed for a long time; RTC, SIP, ICE, etc... The hurdles are social and political.

Here's a decentralised caching proxy I made earlier this year: https://github.com/psybernetics/synchrony

Future plans include a C port (already in the works) and dialing down on the contacts API to bootstrap a three.js CAD tool.

We need handset-to-handset encrypted communication without a server. How good is IPv6 inbound connectivity?

Successfully attacking this problem means progress on numerous fronts: technology, connectivity, bog-simple configuration and operation, recoverability from user or system/hardware errors. Whittling these down over time may work. Most significantly, hardware costs are not presently constraining, limitations lie elsewhere.

Google and Facebook's advertising support strongly tend to centralisation.

Kicking the legs of advertising out from under the stool of publishing might be a means of attacking that particular tendency, which would make a more distributed technical model much more viable.

There's the question of just how distributed you'd want a system to be. There are problems at both low and high levels of centralisation. A fully decentralised model might have a tendency to go rogue, or be subject to petty dictators. It's been interesting to note that even a well-capitalised entity such as the NSA is reported to prioritise specific and generally small values -- single-digit, often single-hand -- of systems for specific attack and interest. There's an argument to be made that strategically weak targets, such as the perennially troubled Yahoo, headed by the morally compromised Ms. Meyer, can be made to crumble where a more commercially robust firm, say, Google, Apple, or Facebook, would be willing to resist. I'm mindful of this.

There's some reason to believe that perhaps a suite of Free Software alternatives would offer more robust installations, and compromises would impose high workfactors, for any given would-be surveillor. The question of creating systems by which data exfiltrations might be more readily detected is another area for exploration.

Going back to the publishing world, there's a long-standing practice of including intentionally fictitious entries within compilations, maps, etc., whose observation in the wild would indicate copying. This might be worth pursuing on a formal basis, particularly through such indicators as financial accounts, email or communications addresses, URLs, etc., extant as canaries which would reveal a data breach.

A huge problem for any widely distributed infrastructure is maintenance and administration. I'm reluctantly concluding that a world in which individuals and households host and administer their own personal data servers isn't viable. I'm not abandoning all hope, but it seems a difficult problem, and one which the Internet of Shit seems determined to prove intractable.

Much of this has to do with the financial underpinnings of the infotech world. As several recent authors have noted (Paul Romer, Jeremy Rifkin, Paul Mason), information-dense goods function poorly in a market-based economic system. They have numerous characteristics which make for poor price discovery and dynamics, with informational asymmetries, heavy up-front (and hence average & fixed) costs, low-to-zero marginal costs, susceptibility to low-cost copying by others (notably China, though this is a practice with ancient traditions -- see Ha-Joon Chang and Frederick List), diminishing marginal returns, long-tail support obligations, and unintended and non-evident consequences (to vendors, users, and bystanders).

A few possible models for a largely-distributed personal informations ervice suggest themselves.

One is the residential-server-as-utility model. Comcast has had such an offering for nearly a decade, that I'm aware of. Essentially, it's a set-top box which can take on additional responsibilities, including home automation and security functions. There are several other utility-type providers who might offer similar capabilities.

There might well be ad-hoc collections of friends or neighbours. A tech-oriented person could easily provide services to hundreds or thousands of others on commodity equipment. The main limitations here are trust and discoverability.

There are arguments for making email into a government-provided service, via the post office. This introduces the risks of government surveillance (already an issue with postal systems), though the protections of legislative restrictions, and public-sector union whistleblower protections, against gross abuse. The fact that physical mail delivery already ensures a brick-and-mortar point of presence in virtually any habitation means that one of the perennial problems of information technology -- establishing, asserting, and recovering identity -- can be achieved through a local visit.

The early Internet spread through a set of social institutions, largely universities. These provided points of access, administration, and accountability to populations of users ranging from a few hundreds to a few tens of thousands per site at major public universities. Whilst this was not an all-encompassing level of provisioning, it is a model of access and social organisation which might be useful to draw on for a more modern implementation.

Public libraries, as an alternative to universities, might offer another option. They already serve as an internet access option for a significant population.

Banks, schools, major retail establishments, and religious centres might be other options.

I'm not sure what exactly will work, but the dynamics I'm pointing at here involve:

Technical capabilities and equipment. A very low bar, and getting lower all the time.

Connectivity. Slightly more difficult, given distribution, land-use, and reliability concerns, but still generally tractable.

Trust. A major factor, especially in a world of eroding social institutions and values. This plays into the dynamics of various systems providers / maintainers quite heavily.

Workforce technical capabilities. Information management skills are remunerative, and could prove difficult to retain. This might change, possibly rapidly. The present market is, however, exceptionally geographically centralised. A distributed node-and-service model might allow for greater flexibility in specific node administrators and technical staff to operate in a much broader choice of areas, including those with lower housing costs than New York City or San Francisco, closer to family or hometowns, etc. I expect some de-skilling of routine obligations, but technology tends to require at least a modicum of fire-fighting capabilities, and perhaps some engineering and planning capacity as well.

Identity and pseudonymity. Both matter. Balancing the risks of data disclosure with those of data loss gets complex. Principle control over data, plus remote backups, ensures against physical loss. Encryption with some model of key escrow, in which a set of parties, several with strong and vested interests in personal privacy can balance privacy against the problem of loss of specific tokens (physical or otherwise).

Ease of use / access. This stuff has to be bog-fucking-simple to provision, deploy, and maintain.

Copyright and copyright compliance. Ultimately, my view is that we're going to have to recognise that attempts to re-bottle the genie, and provide centralised or authorial control over reproductions is doomed. Information-as-public-good strikes me as increasingly inevitable. (Which isn't the same as "inevitable", though it's getting warmer.)

Misbehaviour. People can be asshats. Another reason for forms of key escrow and the like is to allow specific access to specific individuals' information in specific instances, subject to very strong controls.

Search. Traditionally this has relied heavily on data-center based operations. There are distributed alternatives, but none I've seen are particularly adept.

So: yes, challenges, but the fundamental capability (hardware) is well in hand. Spitballing at approaches should eventually see something stick.

> This might be worth pursuing on a formal basis, particularly through such indicators as financial accounts, email or communications addresses, URLs, etc., extant as canaries which would reveal a data breach.

This is a great idea, but it's not clear how many adversaries would do something in an automated fashion that would give away their existence like this.

Apart from the kind of discredited story about Coventry (https://en.wikipedia.org/wiki/Coventry_Blitz#Coventry_and_Ul...), I was just reading somewhere again that there was a policy about Ultra that nothing could be done in response to an Ultra intercept without first creating an alternative cover story -- parallel construction, if you like -- about how the Allies knew about it. (See https://en.wikipedia.org/wiki/Ultra#Safeguarding_of_sources on this point, although I can't remember where I was just reading about it.)

Automatically contacting or attempting to use accounts or URLs mentioned in random data is probably exactly the kind of thing that wouldn't be done by anybody exercising a similar kind of caution today.

I'm actually fairly certain the fictitious entry concept exists in some shape or form. Even name+company@example.com addresses used for email registration are tells of misuse of registration information. Various honeypot traps utilise similar elements. I'm all but certain financial institutions have at least tracking, if not fictitious entry, accounts seeded within various data lists whose use triggers alerts, though I'm not sure those specifically indicate leaks. Document and video/audio tracking have also been used to determine leak sources. So it's not an original idea.

While parallel construction may be used to create an appearance of plausible discovery, an attack against a resource which does not exist would be a strong suggestion that supposedly secret data aren't.

I suspect you're overestimating your adversary. Some will act with extreme reserve. But odds are strong that a curious adversary stumbling across what appears to be public URL or IP + service address, and tickling it will happen with some frequency. Other tells could be constructed of a nontechnical nature -- false documents and such. Behavior consistent with access to that information would be an indication. Audits of non-existent financial accounts, inquiries of non-existent companies or people, etc. The hit rate needn't be perfect, but a single tell would be sufficient.

Huh, now I'm thinking Cliff Stoll did something like this in The Cuckoo's Egg with making up a fake project that intruders then wrote to ask for more information about... or am I misremembering?

Yes, via Wikipedia:

In order to entice the hacker to reveal himself, Stoll set up an elaborate hoax — known today as a honeypot — inventing a fictitious department at LBL that had supposedly been newly formed by an "SDI" contract, also fictitious. When he realized the hacker was particularly interested in the faux SDI entity, he filled the "SDInet" account (operated by the imaginary secretary Barbara Sherwin) with large files full of impressive-sounding bureaucratese. The ploy worked, and the Deutsche Bundespost finally located the hacker at his home in Hanover. The hacker's name was Markus Hess, and he had been engaged for some years in selling the results of his hacking to the Soviet KGB. There was ancillary proof of this when a Hungarian spy contacted the fictitious SDInet at LBL by mail, based on information he could only have obtained through Hess. Apparently this was the KGB's method of double-checking to see if Hess was just making up the information he was selling them.

https://en.m.wikipedia.org/wiki/The_Cuckoo%27s_Egg