I'd dispute the point about the CAP theorem slightly. Thing is, your data - Little Data - is naturally centralized. It doesn't have to be centralized in the same center as everyone else's data though! Then it becomes Big Data, where it's having this giant privacy-violating orgy with everyone else's data all day long.
The amount of traffic that your own Little Data has to serve, unless you're a celebrity in which case you can pay for serious hosting, is never going to require a giant cluster of replicated servers. So CAP just isn't that much of an issue.
I mean seriously... if nodes cannot easily contact each other horizontally then of course everything evolves toward a super-centralized model with large central groups of nodes acting as intermediaries. What part of that is hard to understand?!?
Thanks for the props. A new alpha release of ZeroTier is coming soon, and then it's going into beta with downloadable installers and other nice things. ZT1 is not going to decentralize the 'net, but it does create a lab where people can play with such things. (And it's an interesting VPN alternative for decentralized orgs too.)
I've thought this for a long time. I was even going to cover it in a "computer issues for regular people" book.
EDIT: Cover it in one, not as one, there's a lot more material than that.
Of course most malware and other attacks today bypass the firewall using "pull" based vectors like HTTP and e-mail, but try telling people that. Remotely exploitable "pushable" vulnerabilities are rare these days on stock OSes too, but again try telling people that.
And by firewall in this context I am referring to middle-box firewalls, not local firewalls. The latter are under the control of a box's user/OS and so can easily be opened to permit lateral communication. Middleboxes are the structural culprit here.
Basically, the Internet has spam because identity isn't a limited resource. IP addresses are kind of a limited resource, but they're not really the property of the person using/abusing them, so a blacklist doesn't inflict precise targeted damage, can't be made too draconian, and is easy to evade in lots of ways.
And everything above the IP level is unlimited. If there's an unlimited supply of identities, you can't tell the difference between a new customer and an old enemy. You want the first to have positive default reputation and the second to have infinite negative reputation.
People in the personal cloud community often point to email as proof that spam can be solved. Yes - but spam was solved in email because email already existed in an spam-free Internet. On the Internet we have, there's a much easier solution to the fact that any new protocol which is successful starts to attract spammers. The solution is: stop using the protocol. Google turned off XMPP federation for this very reason.
Basically in an orc-infested environment, you can't have your own cute little bungalow in the cloud. You gotta have an apartment in a giant fortified castle in the cloud.
Having a limited supply of identities, in which identities are (a) property and (b) property you control cryptographically (Bitcoin style, "allodial title"), makes it easy to make spamming not pay, once the price of an identity is greater than the profit a spammer can earn by burning it. And it does not require a central governance authority, or even a central reputation authority. (Reputation authorities shouldn't be built into any system, because if they abuse their own reputations the consequences are insanely dire.)
NAT is a problem, but there are lots of ways to tunnel around it. Which all suck, of course, but...
(1) Every NAT traversal protocol is unique, so there is no interoperability between different apps. The power of IP lies in the fact that it's a lingua franca-- anything can open TCP or send UDP. But anything cannot speak BitTorrent-DHT or Skype or whatever. So there's no potential for exponential growth in capability by tying disparate things together. Firewalls and NAT kill protocol interoperability.
(2) NAT traversal is hard. I know cause I just did it. It's a pain in the rear, and I'm still going to have to build port 80 HTTP tunneling into ZeroTier for that 0.1% of users who cannot use UDP or tunnel through their NAT. So you have to implement NAT-t + a proxy service for everything.
As far as spam goes, you're right. I should have mentioned that. But there are lots of strategies for dealing with spam. Why can't we have a few million small castles instead of one big one, for example? BBSes each had sysops who would kick off abusive users. There are also cryptographic things like hash-cash, Bitcoin economies, trust matrices, etc. These are complex but if we could engineer something good here then it could eventually be packaged into a friendly library that programmers could use without having to understand all the devilish details.
The fact is that we did build a decentralized many-to-many Internet. Then we broke it with firewalls and NAT.
Prediction: one day my protocol will run on top of yours. Rebuilding teh Internets, one layer at a time...
With IPv6 every device could have a first-order IP. We just need to get people to give up middle-box firewalls. That and anyone who attempts to build IPv6 NAT should be placed on trial for crimes against humanity, with the punishment being impalement followed by incineration atop a pyre.
NAT is the devil incarnate. Virtually all evils in the world -- from child abuse to war -- can safely be blamed on NAT.