Decentralising the web: Why is it so hard to achieve?
computing.co.uk
computing.co.uk
What’s easier: building a new internet, or forcing companies like Facebook to be honest and transparent?
Those companies have proved they can't be trusted.
So I believe the answer to that question is to build better, easy to use, decentralized platforms. I think that given a real choice, people would be happy to use open systems. Up until now they haven't been given easily usable alternatives that do all the things they want to do.
The problem is that decentralized systems are always going to be less usable than the centralized systems they compete with. Centralization is what makes better usability possible.
I disagree with this premise.
There's less opportunity for monetary gain, so less effort has been invested in decentralized systems.
I don't think it's an actual inherent technical limitation of decentralized systems. In fact decentralized systems promise many technical benefits. Of course there are also challenges, but in my opinion, not insurmountable ones.
When you let people run away with their creative freedom, you get the Web. That is, nothing talks to anything else in any reasonable way.
And by the way - spam is a problem that those platforms don't solve themselves either as they use their communities to filter that out. So it's only a matter of time before those communities realise they might as well work outside those platforms.
They do so because there is an economic incentive for them.
Decentralized will not do this, as long as they remain non-profit and strive to stay ethical.
This means that the average user will prefer the unethical and addictive platform. Same thing as with fast-food. Not only do decentralized systems need to be technically competitive and surpass hurdles that centralized systems do not have, but they also need users to be made aware of those issues.
They are necessary but utopic.
We need similar campaigns for an ethical internet. People need to understand the problems with the current options and be offered alternatives as good as possible. At the end of the day we can't force companies to behave "ethically"( whatever that means depending on your definition) so a decentralized options will be a better one.
And yet america is fatter than ever and getting fatter quicker than ever.
I think it's time we accept that most people are sheep that give in to even the most primitive forms of manipulation, even after being made aware of it.
Or to put it a bit more delicately, our brain is hardwired to optimise for short term reward functions and the vast majority of us are unable to seek longer term rewards even when consciously desiring them.
So what? That doesn't mean that the campaigns don't work. There has been a decent number of people changing their lives for the better. The fact that there is a tremendous amount of variation on cultural and education levels in the States simply means that certain areas might lag behind in changes. That doesn't mean the changes won't eventually happen.
Companies just provided the hardware.
Perhaps we need to go back to that model.
That's not true. "People" don't care whether the service is centralised or decentralised. They just want the practical benefits of something that looks like a centralised service.
Humanity had built one in just a few decades. Yet, it hadn't managed to successfully enforce any particular human or group behavior in a few millenia. I'm not even really sure humanity has uniform idea about what honesty and transparency actually mean and whenever and how they should be enforced.
I don't think centralisation per se is a defining feature of the platforms you're speaking about.
Last week I gave a talk on the history of decentralisation, or at least my history with it. One of my examples of a successful case of decentralisation is Minecraft vs Lego Universe. Both quite similar games, but Minecraft totally crushed Lego and out-scaled it to an insane degree. As far as I can tell this is largely because Minecraft was much more decentralised (mods, user run servers). In particular Lego Universe was crippled by the cost of centralised content moderation that Mojang simply didn't have at all.
There are examples of decentralised user-owned cooperative type systems outscaling and outperforming centralised alternatives. I'm uncertain to what extent you can make sweeping generalisations ... there certainly seem to be more cases of successful centralised platforms than decentralised, but this is partly because we tend to view decentralisation as an absolute rather than as a spectrum, and partly because a few companies with very strong hiring pipelines have subsidised the creation of vast quantities of centralised software.
You're absolutely, completely, 100% right. This is a critical need!
Sad to say, I can think of several models that have been tried - donations, subscriptions, and merchandising come to mind. None require building a subscription apparatus! This makes them morally superior in very important ways, but all of those approaches come with distinct downsides. They all have distortionary effects on the user experience and condition the continued existence on the service on a handful of people.
Perhaps we should stop and think carefully about why surveillence-fuelled ads are so popular an approach. It's a systems that lets one monetize lurkers who are otherwise minimally interactive. Which is most users. As a result, the income base is much broader and more predictable.
We need better models. And we need to understand our models better.
Well forcing those companies to reform is simply impossible. That's because their incentives are all wrong, so they will always slither around anything you try to get them to do right.
That means that decentralization, as hard as it might be, is always vastly easier than the alternative.
Otherwise, decentralization imposes performance penalties most deem unacceptable for common use. Furthermore, in the case of p2p, you can end up serving illegal content.
The public cannot be confused if you want new technologies to become mainstream. Selling the value of decentralization, something few ordinary people truly care enough to make behavioral or even proactive choices towards adopting, is the wrong strategy currently. That someday will change but it could be a while. The public needs other value propositions like the promise of benefit with little to no effort or mindshare to adopt.
This is similar to why mobile payments faltered (and to some extent still does) in the North American market. Too many players sending out different complex messaging. No unified voice.
Eventually we will eliminate the performance penalties, but mass adoption or demand for a decentralized web will require a complex multi pronged approach that includes education and extensive value adds which are easily understood.
I think that what isn't immediately obvious when most people consider the big silo websites is:
1) The army of moderators that are constantly fighting to remove illegal, offensive, or commercially damaging content. (Twitter might not care about bots posting lies to sway voters, but it doesn't want a reputation that kids are following fake celebrity accounts that post beheading videos every few months).
2) The fact that these sites, by acting as middle men (and mass surveillance infrastructures), get to calculate people's reputation, which is difficult to export between sites in a meaningful way, and is hard to replicate in a decentralised/federated architecture.
The various communities on Steemit all seem to focus rather passionately on protecting their hashtags and feeds from users or content that are abusive or spammy.
The problem is that moderation doesn't scale. Small communities and small instances will nearly always be better at moderation than a large platform, because they can rely on a small number of individuals to get the job done - individuals who are more likely to be consistent, more likely to pay attention to community reactions to posts, and more likely to care in general about what they're doing.
It's difficult for me to think of any large platform (Facebook, Github, Twitter, Reddit, Steam, Amazon, etc...) where moderation and general content quality is better than their smaller counterparts (Mastadon, Itch.io, Gitlab, self-hosted forums, etc...).
Large platforms also become large attack vectors. The same benefits of centralization for users - (content discovery, single sources of truth, and so on) make it very cost-effective and efficient to spam and harass users. With a decentralized system, spammers are less likely to care about whatever small community you're hosting. It's not impossible for them to crawl around the internet spamming everyone, but the cost of doing so is a lot higher than targeting a single platform.
So centralization inevitably leads to large platforms because the market for these platforms is winner-take-all, and it almost a kind of pseudo physical law that large platforms can not be moderated well. I don't want to be absolutist about it, but I'm really having a hard time thinking of even a single exception to that rule.
Maybe Wikipedia, but I'm even kind of doubtful about that - and most of Wikipedia's quality moderation comes from a group of people who are completely obsessed with the project. That mindset also doesn't scale to Twitter/Facebook sizes.
Perhaps there's another comparison to be made? Facebook's moderation, for all its many and myriad faults, is infinitely better than the stuff your average SMTP server sees every day.
Federated decentralized systems are a hell that combines the worst of both centralization and decentralization. It's a great way to offer users the features they want, but at the cost of a set of systems where there's no practical way to stem abuse.
This is great for moderation - Mastadon instances tend to be very focused and to ban bad actors quickly. It's bad for content discovery, because not all of the content is localized in one place.
It's a tradeoff, one that I'm increasingly suspecting is inherent to the debate over centralization, and one that I think might end up being a deciding factor for people choosing which direction they support. If you care about mass indexing and content discovery, maybe you just can't have moderation. If you care about moderation, maybe you just can't have content discovery or mass indexing.
Granted, decentralization is worse for censorship because bad actors can go off and make their own instance, and other people can still subscribe to them. But, censorship != moderation, those are two entirely different concepts that just occasionally overlap with each other.
This is assuming that you're talking about moderation that's more in-depth than just adding a captcha to defeat spammers. With the example you gave of hosting your own fileshare; Facebook has more resources to deal with a certain kind of attack - DOS, XSS, etc... but you're really only talking about a very small subset of attack patterns in that instance. I think federation makes most of those problems go away, because you can standardize purely technical defenses and for the most part they scale just fine across decentralized services.
The moment you start talking about hard problems of moderation like choosing which files you're going to host in the first place, the private SMTP server wins.
When I think about scaling moderation and curbing abuses, I'm not really thinking about the low hanging fruit, I'm thinking about curbing bad actors who are already actively participating on the platform, or who are coming up with novel attacks that don't have any one-size-fits-all solutions.
On the performance side, I think that depends on what's being attempted and how. MaidSafers like to reiterate the notion that popular content will have chunks cached more broadly across the network, but that is only one aspect. There are other things that are not intuitive about how to develop a distributed application with no central authority. And their current model does not include sharing computing power (no Function-as-a-Service type of thing); just storage and bandwidth.
May impose such penalties; can also provide improvements (closer to point-of-use data, greater resiliency etc).
While I am completely pro decentralization (I run my own Nextcloud, XMPP and E-Mail server), I am not very fond of all those 'distributed' ideas. Many of them rely on encryption which might be safe to use today, but encryption has this anoying property becoming unsafe in the future. So puting my data encrypted to some distributed network doesn't feel right for me.
The problem with the tech giants is that their products are so easy to start with that many consumers just use them because they don't see real alternatives. And quite frankly, often there are no equivalent alternatives, as the tech giants throw so much money at their products that very few other products have the same ease of use.
I am not sure about a solution to this dilemma, but I hope that some day we will have more high quality, easy to use, federated services.
One way is to build high quality platforms. Part of why centralised apps tend to be more common than decentralised is that we have pretty good or at least well understood platforms for making centralised apps, but the platforms for making decentralised apps are obscure at best and largely non-existent.
Another way is to narrow and repoint the focus a bit. People thinking about decentralisation tend to immediately gravitate to consumer applications. I've found it easier to get traction in the business-to-business space. Despite all the noise generated by the press consumers often don't really care about things like lockin or data privacy - the tech firms protect people's data well enough, and there isn't usually enough lockin to really worry outside of maybe smartphone operating systems. However businesses are often much more sensitive to these issues. Yes they will outsource and use centralised services too, but they're much more likely to care about things like confidentiality, privacy of their data/trade secrets, they sometimes have regulatory constraints that push them in the direction of decentralisation etc.
So that's one reason why I'm working on Corda, which is a platform for p2p business. It tends to get described as a blockchain system and it's got a lot of tech inspired by Bitcoin but I see it more as a platform for building decentralised apps (or more accurately, apps where you can precisely distribute and control trust points in accordance with your needs).
Better platforms are absolutely the way to go for consumer apps as well. Unfortunately building such platforms will probably involve sacrificing some sacred cows. For instance the web is a terribly centralising platform. It's great at decentralising information and documents but when (ab)used as a way to build ordinary applications it creates far more centralisation than we've ever seen before. To get decentralised apps becoming commonplace we probably need to tackle the dominance of HTML5 and HTTP as a (desktop/laptop) app platform.
If you have a typical-sized online service, I think you can be in the same league as the tech giants with two full-time employees focused on ease of use. With four I think you can be on equal footing. And if you charge users for your service, you can focus on ease-of-use improvements that let users get things done faster. A tech giant whose revenue increases with "engagement" is going to make few improvements that help people leave their site faster.
EDIT: To be clear, I'm talking about competing on ease of use, not feature count.
The article seemed well done, the problem (from a information theory / network graph) though seemed weighted towards blockchain-like assumptions. This is fine, as more information needs to be seeded on this.
For those interested in the title's question:
Coordination might seem like the hard problem, but it is only hard because the majority of us in this community made it hard by using immutable data structures. If we use mutable data structures (you know my bias, but it is possible with https://github.com/amark/gun , proof: Mitra at Internet Archive integrated it in 1 week, now decentralized IA runs on GUN), we get some other problems but they are less than O(log N) in complexity.
Here is why coordination on immutability is hard:
O(N^2)
Just sit and think about it.
Intuitively it makes sense:
If I have an index to make something fast, and if it is immutable.
When I "update" the index, I have to create a new index.
Now the old index can't find the new index.
Therefore, I need to index the index, etc.
Repeat.
Therefore, as long as the web never changes, a decentralized immutable web will eventually be fast, but it must be true that if while the decentralized web changes, it will be hard.
This is not true if you can do decentralized mutable data. Check out our work! I hope this comment added novel insight to your day. :) If it did, let me buy you coffee next time you are in town. Cheers!
At the end of the day the average user just wants a convenient and reliable way to reach content and information. Centralization has been the best way to serve it, and I don't think it will change.
Most people were on dialup in Napster's heyday, so I don't think that can be the issue. Music files are small.
I feel most people would jump ship if there was a viable, easy to use, alternative where you had ownership of your own data.
It's wrong that people don't understand or want to not depend on something like Facebook. At least where I am, a lot of people understand the issues. But there's nothing really like it, and life in my city is quite difficult without.
It's nothing that can't be fixed though! I think a project like Pixelfed (https://pixelfed.org/) is doing things right. We can get there.
Except e-mail and HTTP.
> We need a more product-oriented approach.
I think this is how we landed in the current situation. Instead of protocols (IRC, XMPP), you get products (Slack, Hipchat), which are walled gardens. I wish there was a way to incentivize companies to stick to shared protocols, instead of creating their own.
The general public is just not very interested in the implementation details of technical products, as long as they work well enough and don't cost a lot. If you really want incentivize companies to implement open protocols, find a way for companies to make more money with the open protocol than with the walled garden, because companies is where most of the engineers are and they are the ones that will make it if the incentive is there.
These days I'm more interested in solutions where companies produce decentralised systems, and they provide the leadership and drive necessary, with the right safeguards in place to allow their leadership to be removed and replaced if necessary. Rather than the more traditional approach of trying to not have any leadership, or do it all by committee.
This caused problems whenever anyone moved so in the early days centralised services were seen as a solution such as Hotmail and Geocities. No one in 1995 realised how it would turn out into abusive monopolies...
The same model could work now, with a small difference: instead of storing only data, it may also host applications that manage those data, and collaborate with same/other applications of same/other users.
For example, a photo app could allow us to manage our photos at a specific host of our choice; the same photo app uses a sharing protocol so as that the photos are shared with other users of our choice.
The stored data are encrypted by our own keys and therefore not visible to third parties, we still get to share our stuff with our mates, it's decentralized because no central authority can lay claim on our data, it's protected because it is encrypted for us, but the infrastructure required for this to run is provided by a centralized entity.
Bitcoin wallets for example, suffer a lot because the average user doesn't have much of an idea that they need a strong password or what these random numbers mean other that they might behave like bank accounts.
The ideal decentralized UI/UX will work largely by copy paste, with human readable names (@me@mastodon.social, for example) and doesn't have too many buttons that need more than an icon or single word to explain them.
On top of that it also needs to look modern and neat while being fast and snappy on mobile devices without sucking up battery or limiting how the users can interact (ie, not possible to share photos directly with the app is a no-go)
These design limitations are solved more easily by centralized services as they can just throw money at it.
Urbit's conceptual path to decentralisation seems sound: build a new friendly network on top of the internet, and build a new friendly server OS on top of Unix.
Even for services where I have to run my own server, I rent a VPS at some hosting facility rather than maintaining my own hardware or even renting bare metal. It simply doesn't make sense to run my own.
And if I bothered to have a web presence, again, I'd host it somewhere. It simply doesn't make sense to maintain your own except as a hobby.
Moreover, all the blockchain stuff looks to me like a solution looking for a problem instead of something generally useful. Yes, it will make sense for decentralized payments if the exchange rates ever stabilize. No, adding blockchain to everything won't help.
Hosting your website with a hosting service has no network effect for the hosting company. If they misbehave, you can simply move to a different hosting company with absolutely no change visible to the outside. You keep your domain, you are still reachable using the same web browsers at the same URI, via the same bookmarks, via the same hyperlinks.
So, while that arguably is more centralized than running your own hardware at home, it's nowhere near the level of centralization of power with services that have a business model of building a monopoly, such as Github or Facebook. If Facebook mistreats you, you cannot leave Facebook and still keep using the service they are offering.
Especially with the example of git hosting, the primary problem isn't the hosting part, it's the specific business model of Github. The problem isn't that you are delegating the hosting to a specialized business. The problem is that you are delegating it to a business that's holding your hostage and using you as leverage against the general public.
Also, you are making some assumptions that might be worth questioning. Like, why doesn't it make sense to maintain your own web server? I assume you assume that takes a lot of tinkering that you don't want to bother with. But why would that be the case? Is that a technical necessity? Or is that just some experience of yours that doesn't really have to be that way? Is there any reason why hosting your own server couldn't be more reliable and less work than using a hosting service?
One example is Github, a centralized service that makes it easier for teams to use a source control system that was originally designed to be decentralized and distributed. When it was purchased by Microsoft and people started looking elsewhere, they looked to competing centralized solutions like GitLab and Bitbucket.
Given that centralization is so useful, and people want it so much, it should not be resisted. We would be better off if the effort being put into decentralization was instead put into better centralized services that respect user privacy.
One example is Microsoft, a monopolistic company that makes an operating system that everyone uses, replacing lots of unpopular incompatible ones.
Given that monopolies are so useful, and people want them so much, they should not be resisted. We would be better off if the effort being put into making markets competitive was instead put into increasing Microsoft's share price.
It's also not sustained by VC cash anymore -- Microsoft is buying the company.