Hacker News Reader

Comment by EGreg | Hacker News Reader

Am I the only one who is worried about "the cloud" phenomenon centralizing a lot of the internet under the control of a few companies?

I think it mostly has to do with the client-server technology of the web today. Once effective technology exists to decentralize key aspects of the platform - hosting, accounts, etc. - you'll see less consolidation in one provider. You'll also see a lot less feudalism when it comes to the power dynamics of users and organizations vs services (like Uber, Facebook, Google, etc.)

There needs to be a protocol to decentralize social the way bitcoin decentralized money.

Disclaimer: I'm working on just such a protocol and reference implementation (http://qbix.com/platform/features/distributed)

View on HN

Although Amazon, Google and Microsoft are the big players, everyone and their dog are implementing clouds.

There's even an open source cloud: https://www.openstack.org/ This is being used by some large companies (e.g. HP), and can also be used by companies who want to run a VM setup on their own hardware like they probably already do with VMware, but want a cloud-like API for it.

Personally, I think the consolidation down to a few players is a big concern.

At the moment, there's a fair amount of diversity out there for things like:

- server hardware, network hardware, etc

- specific linux distribution

- webserver "brand" (apache, nginx, etc) and version

- database choices (oracle, postgres, mysql, etc) and versions

- application server level (java/j2ee/tomcat, node, python, php, etc)

I would argue that most of that diversity is currently driven by non-cloud applications. You do, of course, have choices in the cloud as well. However, they all have a lower friction path that would naturally consolidate the current broad array of choices down to fewer solutions.

The hard bit is trying to visualize if the majority of non-cloud solutions are eventually going to move to the cloud, or will there always be a significantly large non-cloud solution set out there? There's also the question of how many of the cloud customers would leverage the providers' default tech stack choices versus customers making their own choices.

I think it's a concern, but probably not as great as you might perceive. Centralization is fine (though maybe not great), as long as it is not forced. As long as the web remains open and free from a governance point of view, monopolies can only exist as far as one company can offer superior technology and offerings in general. If they fail (and thus that centralized node fails), as long as no one is 'forced' to stay with that centralized cloud provider, they can switch to competitors. Furthermore, in the future I can only see switching hosting providers becoming easier and easier to do, especially with services such as Docker becoming popular.

>Once effective technology exists to decentralize key aspects of the platform

This has been promised, at least, since the 90s. So far its really not happening. If anything the exact opposite is.

>There needs to be a protocol to decentralize social the way bitcoin decentralized money.

In the currency wars, bitcoin isn't even a player. Its quite the niche, like FB alternative diaspora*. Heck, by now we should all have local copies of wikipedia and only do nightly diffs right? Everyone wants their own personal decentralized copy of wikipedia, right?

I think centralization is not only going to be popular due to cost savings and ease of management, but the natural way to go for information technology. Complex systems often consolidate balkanized parts into larger wholes. This is also why we have ~200 countries instead of tens of thousands of city-states. Or why so many markets in technology break down into natural monopolies/duopolies.

The network effect, branding, lock-in, and minimizing customer confusion are fairly significant forces here on the customer end. Not to mention, the ability for large organizations to buy up smaller ones and also having the financial resources to soak up all the hot talent. From the abuse end of things, there's defeating smaller competitors with crony capitalism like patent abuse or predatory pricing, both of which larger entities have an easier time of doing.

Say that to a libertarian lol. Especially an anarcho-capitalist.

I actually see this as a consequence of technology. Before git, version control was mostly centralized. Before Linux, WebKit and open source explosion, the OSes and browsers were manufactured by one or two main companies, and compared to what we expect now, the IE monopoly and other siloed monopoly products seemed terrible. I predict facebook will seem that way in the future when we have choice where to host our own social node, and which plugins to install.

There is a lot of power imbalance with centralization. The consolidation of telcos led to the net neutrality debate, but the problem is central control by huge players. Amazon can squeeze publishers. Apple rules app developers with an iron fist. Twitter caused a ruckus knocking companies off their playform whenever they felt like it. Google can disconnect a website and have its traffic plummet. Etc etc.

I think that when the technology exists, decentralization helps the consumer, through competition and raising their expectations.

Except your argument doesn't work well because we're centralizing FOSS. (perhaps I am misunderstanding your argument?) So you still have linux, webkit, etc. You just get linux nowadays via AWS, Linode, etc. And you get Webkit via Apple or Google.

Its not one or the other. These two systems can work hand-in-hand. FOSS commingling with commercial centralization has been quite the success story. FOSS on its own as PoC's or nerd toys chatted about in IRC isn't enough to get real market penetration.

I don't see it that way. We don't get WebKit through Google and Apple. We get long tail contributions to the project:

https://www.ted.com/talks/clay_shirky_on_institutions_versus...

What we have is centralization in phone manufacturers when it comes to Apple, but not Android. Google tried to re-centralize by making the Play Store but Amazon and others already forked Android. In fact, Chrome is a fork of WebKit which Apple started for Safari! And that is how we had such an acceleration of features. Open source and commercial competition.

I don't deny FOSS commingles with commercial offerings. Rather I'm saying that when platforms became freely forkable, the quality AND the freedom rose dramatically. That has yet to happen in social -- as you say, Diaspora* didn't take off.

But once there is a viable FOSS social tech platform, it will disrupt all the big players. Who needs http://twitter.com/MyBrand when I can just add the Twitter module to mybrand.com !

You can also see the problems with Facebook campaigning for India to have a billion people have their internet signal go to Facebook's server farms. Besides state spy agencies, this is also simply bad engineering. Early internet apps like email worked even if email.com was down. I could use the local network.

I think we will actually soon have software that runs on social mesh networks and that's going to enable everyone in the world to share files, communicate and collaborate faster and wasting less energy than going through a central hub. You'll see a re decentralization of the internet in the next several years.

There isn't much of a long tail to browser development these days, the build requirements are too crazy. And the identi.ca crowd has been around from the beginning, with no more uptake than they ever had.

Im a pretty hard core libertarian and completely agree with what both you and the parent commenter said.

In a capitalistic market, natural monopolies and oligarchies will form. So long as competitors can enter (i.e. arent legislated out) of the market there's nothing wrong.

Similarly, forcing businesses that work better at scale to become smaller will increase prices and decrease efficiency. Obviously, this is a somewhat idealized situation, so in reality it's going to be somewhere in between (because companies, governments, and people are all willing to not play "fair" in a market). Being a libertarian, I just error on the side of letting the market decide, as opposed to legislators deciding how the market should be.

Yep. I just see cities and states as a special case of organizations. And the same central planners you're talking about exist also in corporations, unions and other organizations. If you want to let the market drive innovation, the same forces apply within these organizations: if the platform is freely installable by anyone and the barrier to enhancement is low, then it leads to better quality and more freedom for everyone.

The same reasons that most people prefer to "just use" gmail, facebook, amazon etc. are the same reasons that people prefer to "just live in" a state and not figure everything out themselves. So the libertarians who envision a world where people pay competing rating agencies, security agencies, firefighting companies etc. face the same challenge as those who want people to use OSS software instead of hosted platforms developed and supported by big companies. The smaller stuff will always be more rinkydink, but the competition will drive expectations of freedom and quality up for everyone, even those using the states / corporations.

One of the other commenters already pointed out services like Docker are letting us abstract away from the specifics and just run the software wherever there's an OS that'll take it. That reduces the hold any one provider can have and forces them to compete on price / features rather than technology lock-in. If one of them implements a requirement you must use their standard APIs for XYZ, then you're probably going to get a mass exodus.

Longer term, I think Amazon's cloud will continue to eat market share, especially as an entirely new generation of techies comes online whose only experience has been working with AWS and other cloud providers. Kind of like the transition from desktop software in the 80s/90s to server-based web apps in the 90s-early 00s. If you're building a company / app today, the first move is to go to the cloud and get on AWS. That's the "safe" business move now rather than spending capital to buy hardware, getting a sysadmin, etc. etc. etc.

For the market, I'm hoping we see someone develop a set of common APIs that abstract away common specifics across AWS/Google Cloud/Azure and let you run a generic web stack and dynamically move parts of your app around based on pricing. If Google drops below AWS and your demand is spiking, you setup new instances on Google, rather than spinning up AWS instances, and use a micro service API to interact with the database. Behind the scenes, the API layer tracks the different clouds you configured for and dynamically moves most pieces of your application except data storage as your target conditions change. I envision spot cost, demand, latency, and host availability / provisioning time all playing a role in this cost algorithm. Eventually you end up with agnostic applications that don't ever live anywhere. They migrate and dynamically adjust across different clouds turning each individual cloud in part of a larger, more agnostic cloud for some services.

Looking at the datacenter locations for AWS, Azure, GCE, Linode, softlayer and DO, it'd be challenging to develop reliable multi-cloud datacenter agnostic apps with any consistent backend datastore requirements. In the end, you may get better availability and performance if you do things carefully, but I think the cost savings won't be there unfortunately.

Walmart is releasing OneOps. It's supposedly their internal platform to be cloud-provider agnostic. I've not used it (yet) and haven't heard of anyone else using it but it sounds like a step in that direction.

http://www.walmartlabs.com/2015/10/walmartlabs-oneops-open-s...

That is the one hiccup I foresee happening. The best current workaround I could think of would be to setup your datastore as a micro service the other parts of the app could interact with. That does lock your datastore into a single provider, but gives you flexibility with the rest of the app.

> For the market, I'm hoping we see someone develop a set of common APIs that abstract away common specifics across AWS/Google Cloud/Azure and let you run a generic web stack and dynamically move parts of your app around based on pricing.

At an IAAS level, you can use BOSH to manage stateful systems AWS, OpenStack, vSphere, Azure and GCE. The last two are basically new contributions by Microsoft and Google.

Which matters because once you have BOSH, you've done a lot of the heavy lifting required to install Cloud Foundry.

We are not at the point where there is seamless multi-provider support. But it's a foreseeable capability. We see a lot of capabilities, because of the flexibility of the underlying components of Cloud Foundry -- particularly BOSH ( low-level system deployment and liveness), Diego (container placement, replacement and observation) and Garden (container runtime).

My usual disclaimer: I work for Pivotal, which donates the majority of engineering effort to Cloud Foundry. I should doubly add that saying "it could plausibly do thing X" doesn't mean I'm able to commit us to it.

You can also abstract the network (and work around some of your provider's limitations e.g. IPv6) with an overlay network.

We've done some POCs around integrating an agent inside Docker containers and it works like a charm, too. Multi host networking for containers, regardless of your provider's networking limitations :)

If anyone's curious: https://wormhole.network

i'm betting intel is starting to get concerned. pretty soon they're only going to have a handful of customers, who can then dictate pricing.

Does it make any difference from their point of view if their customers are server manufacturers or cloud providers?

Depends on whether they're running Intel or not...

It makes a difference if they're dealing with a disproportionate amount of revenue concentrated in the hands of only a couple major customers. If the customers end up having disproportionate control over revenue, they can use that bully pulpit to force Intel to make changes the rest of the market may not want, hit certain price points that Intel wouldn't otherwise, etc. using the threat of do it or we go to AMD/ARM/IBM Power/build our own chip.

Edit: Fixed wording.

>or we go to AMD/ARM/IBM Power/build our own chip.

Given the difficulty Intel has seen getting the new E5 and E7 chips out I personally don't see any of the others as competition when it comes to producing a power efficient X86-64 chip for the DC at scale.

It's possible Intel keeps a stranglehold on the DC due to DC providers wanting X86-64 compatible chipsets and they do have a sizable advantage in their manufacturing process at this point.

I think the greater danger isn't other X86-64 chip manufacturers though. The only real competition is AMD/ATi + GlobalFoundry and their chipset process is a generation or two behind. The greater danger is the market moving to an entirely new architecture. Think about ARM in the datacenter. Less power consumption, smaller boxes, less cooling requirements. DCs could make a strong argument it's cost efficient for them to jump and if a major one jumps, developers will probably follow. Most of what we build today uses a software stack built on top of a managed language (Java/Python/Ruby) and doesn't rely on being native compiled. It reduces the opportunity cost of switching.

... Or MIPS don't forget.

Lots of cloud migration/workload tools out there - OneOps (by WalmartLabs) is one example. Check out http://oneops.github.io/admin/getting-started/#installing-va...

Have you ever been under the control of a central corporate infrastructure group? I've had good and experienced the worst as well. I'll gladly take my chances with AWS instead of dealing with internal groups.

My main concern is the security angle, as the "Big 3" are all pushing for a barrier across country lines. I'd prefer any of the data I store in the cloud to stay in a US cloud.

Reply on news.ycombinator.com