How Long Can AWS Keep Climbing Its Steep Growth Curve? | Hacker News Reader

How Long Can AWS Keep Climbing Its Steep Growth Curve?

nextplatform.com

57 pointsjonbaer43 comments

I happen to spend a fair amount of time in and around non-tech Fortune 500 companies. Almost all of them still have either their own data centers, or a "we'll run your mess for less" type contract (non-cloud) with companies like Rackspace, HP, etc. I suspect the same applies for government agencies at all levels.

I would guess that a fair amount of the potential cloud growth is in migrating all of that to the cloud, either as an actual migration, or as some new solution built in a cloud that retires an in-house hosted solution.

The eventual consolidation of all of that into a pretty small number of big players (Amazon/Google/Microsoft) is a bit concerning in that the reduced diversity of solutions opens up some security, reliability, competitive cost, etc, concerns.

>I suspect the same applies for government agencies at all levels.

I wonder if AWS has made any headway after their initial CIA contract making them a private AWS instance.

It's called GovCloud. AWS has obtained FISMA certification for use as a government provider.

AFAIK, the CIA uses a hybrid setup with the government hosting very large data centers of their own for data and AWS being used to build front-ends that access that data.

It's important to consider the sheer scale of government resources. They run at least one of their own private cloud providers. They're just not accessible to the 'outside' world.

> I would guess that a fair amount of the potential cloud growth is in migrating all of that to the cloud, either as an actual migration, or as some new solution built in a cloud that retires an in-house hosted solution.

Depends on the workload. If your workload is predictable, and you don't need "scale to infinity on a dime", buying your own hardware and amortizing over 3-5 years is typically much cheaper (~40% savings, roughly) than what AWS offers.

Amazon seems to be having success selling AWS as a landing point for things that are predictable and don't need instant scaling.

Edit: The context here is about non-tech Fortune 500 companies, Government Agencies, etc...and the very large amount of existing "non cloud" opportunity they present for the likes of Amazon. Think General Mills, American Airlines, CVS, Kroger, etc.

For example: https://aws.amazon.com/solutions/case-studies/dole/

Sure, it's odd they think running Sharepoint on AWS is saving them money, but they moved :)

Lots of people sell things people don't need. There's something to be said for marketing.

Github and Stackoverflow, to name two that come to mind, run on their own hardware in a datacenter. Backblaze built out its own storage system in a datacenter due to how much cheaper it was than AWS' S3 system.

I think AWS is a wonderful system to prototype in, before you move to your own gear if you're profitable. But every dollar you spend in AWS is gone once its spent; with new US Section 179 deduction rules, you can immediately write off $500K/year in hardware costs. That means that you own that hardware forever. That might not matter to a Fortune 50 or 500 business, but it matters a lot to small to midsized businesses.

Disclaimer: I do devops/infrastructure, and love AWS, but also don't believe in spending more than you must for a solution.

And if it's working fine, or you are abit tight for cash, you can stretch it out a couple years more if you own it. Not so if you are renting, and that goes for software too. Looking at you Adobe.

150 years ago, everyone generated their own energy. 100 years ago, most larger users still generated their own energy. Over time, almost everyone migrated "to the grid" because the centralized power companies did it better, cheaper and more reliability.

I suspect the same will happen with computing. At this point most of the smaller players (startups) are using AWS or another cloud provider. Some of the more bleeding edge larger companies are doing it too. Eventually even the largest companies will move "to the cloud", leaving only the very largest companies (who incidentally still generate their own energy even today) running their own servers.

And 15 years ago, more and more bought photovoltaic cells to generate their own energy and recently buy electric cars. It's probably a cycle.

We had mainframes with timesharing OS and terminals. Then with desktop PCs everyone got a computer. With cloud computing and video streaming we are back to centralised computers. On the otherside there are more and more who use peer-to-peer software, blockchain technologie and want localized solutions. It's probably a cycle.

you're talking about the power grid like it's some monolithic entity like amazon -- it's not. every region has its own, and sometimes multiple, power companies. some are public. LA DWP coexisting with edison is a good example of this.

there's also a high degree of regulation to make sure this stuff works together. see: rolling blackouts.

Oh I wasn't saying it would all be Amazon. I'm saying it will all be 3rd party clouds. The number of providers is up for grabs.

i certainly hope so. amazon is looking to be the next microsoft of the 90s.

i think most people are just too young to remember how horrendously bad it is when one company owns most of the market.

Am I the only one who is worried about "the cloud" phenomenon centralizing a lot of the internet under the control of a few companies?

I think it mostly has to do with the client-server technology of the web today. Once effective technology exists to decentralize key aspects of the platform - hosting, accounts, etc. - you'll see less consolidation in one provider. You'll also see a lot less feudalism when it comes to the power dynamics of users and organizations vs services (like Uber, Facebook, Google, etc.)

There needs to be a protocol to decentralize social the way bitcoin decentralized money.

Disclaimer: I'm working on just such a protocol and reference implementation (http://qbix.com/platform/features/distributed)

i'm betting intel is starting to get concerned. pretty soon they're only going to have a handful of customers, who can then dictate pricing.

Does it make any difference from their point of view if their customers are server manufacturers or cloud providers?

Depends on whether they're running Intel or not...

It makes a difference if they're dealing with a disproportionate amount of revenue concentrated in the hands of only a couple major customers. If the customers end up having disproportionate control over revenue, they can use that bully pulpit to force Intel to make changes the rest of the market may not want, hit certain price points that Intel wouldn't otherwise, etc. using the threat of do it or we go to AMD/ARM/IBM Power/build our own chip.

Edit: Fixed wording.

>or we go to AMD/ARM/IBM Power/build our own chip.

Given the difficulty Intel has seen getting the new E5 and E7 chips out I personally don't see any of the others as competition when it comes to producing a power efficient X86-64 chip for the DC at scale.

It's possible Intel keeps a stranglehold on the DC due to DC providers wanting X86-64 compatible chipsets and they do have a sizable advantage in their manufacturing process at this point.

I think the greater danger isn't other X86-64 chip manufacturers though. The only real competition is AMD/ATi + GlobalFoundry and their chipset process is a generation or two behind. The greater danger is the market moving to an entirely new architecture. Think about ARM in the datacenter. Less power consumption, smaller boxes, less cooling requirements. DCs could make a strong argument it's cost efficient for them to jump and if a major one jumps, developers will probably follow. Most of what we build today uses a software stack built on top of a managed language (Java/Python/Ruby) and doesn't rely on being native compiled. It reduces the opportunity cost of switching.

... Or MIPS don't forget.

Although Amazon, Google and Microsoft are the big players, everyone and their dog are implementing clouds.

There's even an open source cloud: https://www.openstack.org/ This is being used by some large companies (e.g. HP), and can also be used by companies who want to run a VM setup on their own hardware like they probably already do with VMware, but want a cloud-like API for it.

I think it's a concern, but probably not as great as you might perceive. Centralization is fine (though maybe not great), as long as it is not forced. As long as the web remains open and free from a governance point of view, monopolies can only exist as far as one company can offer superior technology and offerings in general. If they fail (and thus that centralized node fails), as long as no one is 'forced' to stay with that centralized cloud provider, they can switch to competitors. Furthermore, in the future I can only see switching hosting providers becoming easier and easier to do, especially with services such as Docker becoming popular.

>Once effective technology exists to decentralize key aspects of the platform

This has been promised, at least, since the 90s. So far its really not happening. If anything the exact opposite is.

>There needs to be a protocol to decentralize social the way bitcoin decentralized money.

In the currency wars, bitcoin isn't even a player. Its quite the niche, like FB alternative diaspora*. Heck, by now we should all have local copies of wikipedia and only do nightly diffs right? Everyone wants their own personal decentralized copy of wikipedia, right?

I think centralization is not only going to be popular due to cost savings and ease of management, but the natural way to go for information technology. Complex systems often consolidate balkanized parts into larger wholes. This is also why we have ~200 countries instead of tens of thousands of city-states. Or why so many markets in technology break down into natural monopolies/duopolies.

The network effect, branding, lock-in, and minimizing customer confusion are fairly significant forces here on the customer end. Not to mention, the ability for large organizations to buy up smaller ones and also having the financial resources to soak up all the hot talent. From the abuse end of things, there's defeating smaller competitors with crony capitalism like patent abuse or predatory pricing, both of which larger entities have an easier time of doing.

Say that to a libertarian lol. Especially an anarcho-capitalist.

I actually see this as a consequence of technology. Before git, version control was mostly centralized. Before Linux, WebKit and open source explosion, the OSes and browsers were manufactured by one or two main companies, and compared to what we expect now, the IE monopoly and other siloed monopoly products seemed terrible. I predict facebook will seem that way in the future when we have choice where to host our own social node, and which plugins to install.

There is a lot of power imbalance with centralization. The consolidation of telcos led to the net neutrality debate, but the problem is central control by huge players. Amazon can squeeze publishers. Apple rules app developers with an iron fist. Twitter caused a ruckus knocking companies off their playform whenever they felt like it. Google can disconnect a website and have its traffic plummet. Etc etc.

I think that when the technology exists, decentralization helps the consumer, through competition and raising their expectations.

Except your argument doesn't work well because we're centralizing FOSS. (perhaps I am misunderstanding your argument?) So you still have linux, webkit, etc. You just get linux nowadays via AWS, Linode, etc. And you get Webkit via Apple or Google.

Its not one or the other. These two systems can work hand-in-hand. FOSS commingling with commercial centralization has been quite the success story. FOSS on its own as PoC's or nerd toys chatted about in IRC isn't enough to get real market penetration.

I don't see it that way. We don't get WebKit through Google and Apple. We get long tail contributions to the project:

https://www.ted.com/talks/clay_shirky_on_institutions_versus...

What we have is centralization in phone manufacturers when it comes to Apple, but not Android. Google tried to re-centralize by making the Play Store but Amazon and others already forked Android. In fact, Chrome is a fork of WebKit which Apple started for Safari! And that is how we had such an acceleration of features. Open source and commercial competition.

I don't deny FOSS commingles with commercial offerings. Rather I'm saying that when platforms became freely forkable, the quality AND the freedom rose dramatically. That has yet to happen in social -- as you say, Diaspora* didn't take off.

But once there is a viable FOSS social tech platform, it will disrupt all the big players. Who needs http://twitter.com/MyBrand when I can just add the Twitter module to mybrand.com !

You can also see the problems with Facebook campaigning for India to have a billion people have their internet signal go to Facebook's server farms. Besides state spy agencies, this is also simply bad engineering. Early internet apps like email worked even if email.com was down. I could use the local network.

I think we will actually soon have software that runs on social mesh networks and that's going to enable everyone in the world to share files, communicate and collaborate faster and wasting less energy than going through a central hub. You'll see a re decentralization of the internet in the next several years.

There isn't much of a long tail to browser development these days, the build requirements are too crazy. And the identi.ca crowd has been around from the beginning, with no more uptake than they ever had.

Im a pretty hard core libertarian and completely agree with what both you and the parent commenter said.

In a capitalistic market, natural monopolies and oligarchies will form. So long as competitors can enter (i.e. arent legislated out) of the market there's nothing wrong.

Similarly, forcing businesses that work better at scale to become smaller will increase prices and decrease efficiency. Obviously, this is a somewhat idealized situation, so in reality it's going to be somewhere in between (because companies, governments, and people are all willing to not play "fair" in a market). Being a libertarian, I just error on the side of letting the market decide, as opposed to legislators deciding how the market should be.

Yep. I just see cities and states as a special case of organizations. And the same central planners you're talking about exist also in corporations, unions and other organizations. If you want to let the market drive innovation, the same forces apply within these organizations: if the platform is freely installable by anyone and the barrier to enhancement is low, then it leads to better quality and more freedom for everyone.

The same reasons that most people prefer to "just use" gmail, facebook, amazon etc. are the same reasons that people prefer to "just live in" a state and not figure everything out themselves. So the libertarians who envision a world where people pay competing rating agencies, security agencies, firefighting companies etc. face the same challenge as those who want people to use OSS software instead of hosted platforms developed and supported by big companies. The smaller stuff will always be more rinkydink, but the competition will drive expectations of freedom and quality up for everyone, even those using the states / corporations.

One of the other commenters already pointed out services like Docker are letting us abstract away from the specifics and just run the software wherever there's an OS that'll take it. That reduces the hold any one provider can have and forces them to compete on price / features rather than technology lock-in. If one of them implements a requirement you must use their standard APIs for XYZ, then you're probably going to get a mass exodus.

Longer term, I think Amazon's cloud will continue to eat market share, especially as an entirely new generation of techies comes online whose only experience has been working with AWS and other cloud providers. Kind of like the transition from desktop software in the 80s/90s to server-based web apps in the 90s-early 00s. If you're building a company / app today, the first move is to go to the cloud and get on AWS. That's the "safe" business move now rather than spending capital to buy hardware, getting a sysadmin, etc. etc. etc.

For the market, I'm hoping we see someone develop a set of common APIs that abstract away common specifics across AWS/Google Cloud/Azure and let you run a generic web stack and dynamically move parts of your app around based on pricing. If Google drops below AWS and your demand is spiking, you setup new instances on Google, rather than spinning up AWS instances, and use a micro service API to interact with the database. Behind the scenes, the API layer tracks the different clouds you configured for and dynamically moves most pieces of your application except data storage as your target conditions change. I envision spot cost, demand, latency, and host availability / provisioning time all playing a role in this cost algorithm. Eventually you end up with agnostic applications that don't ever live anywhere. They migrate and dynamically adjust across different clouds turning each individual cloud in part of a larger, more agnostic cloud for some services.

Looking at the datacenter locations for AWS, Azure, GCE, Linode, softlayer and DO, it'd be challenging to develop reliable multi-cloud datacenter agnostic apps with any consistent backend datastore requirements. In the end, you may get better availability and performance if you do things carefully, but I think the cost savings won't be there unfortunately.

Walmart is releasing OneOps. It's supposedly their internal platform to be cloud-provider agnostic. I've not used it (yet) and haven't heard of anyone else using it but it sounds like a step in that direction.

http://www.walmartlabs.com/2015/10/walmartlabs-oneops-open-s...

That is the one hiccup I foresee happening. The best current workaround I could think of would be to setup your datastore as a micro service the other parts of the app could interact with. That does lock your datastore into a single provider, but gives you flexibility with the rest of the app.

> For the market, I'm hoping we see someone develop a set of common APIs that abstract away common specifics across AWS/Google Cloud/Azure and let you run a generic web stack and dynamically move parts of your app around based on pricing.

At an IAAS level, you can use BOSH to manage stateful systems AWS, OpenStack, vSphere, Azure and GCE. The last two are basically new contributions by Microsoft and Google.

Which matters because once you have BOSH, you've done a lot of the heavy lifting required to install Cloud Foundry.

We are not at the point where there is seamless multi-provider support. But it's a foreseeable capability. We see a lot of capabilities, because of the flexibility of the underlying components of Cloud Foundry -- particularly BOSH ( low-level system deployment and liveness), Diego (container placement, replacement and observation) and Garden (container runtime).

My usual disclaimer: I work for Pivotal, which donates the majority of engineering effort to Cloud Foundry. I should doubly add that saying "it could plausibly do thing X" doesn't mean I'm able to commit us to it.

You can also abstract the network (and work around some of your provider's limitations e.g. IPv6) with an overlay network.

We've done some POCs around integrating an agent inside Docker containers and it works like a charm, too. Multi host networking for containers, regardless of your provider's networking limitations :)

If anyone's curious: https://wormhole.network

Have you ever been under the control of a central corporate infrastructure group? I've had good and experienced the worst as well. I'll gladly take my chances with AWS instead of dealing with internal groups.

Personally, I think the consolidation down to a few players is a big concern.

At the moment, there's a fair amount of diversity out there for things like:

- server hardware, network hardware, etc

- specific linux distribution

- webserver "brand" (apache, nginx, etc) and version

- database choices (oracle, postgres, mysql, etc) and versions

- application server level (java/j2ee/tomcat, node, python, php, etc)

I would argue that most of that diversity is currently driven by non-cloud applications. You do, of course, have choices in the cloud as well. However, they all have a lower friction path that would naturally consolidate the current broad array of choices down to fewer solutions.

The hard bit is trying to visualize if the majority of non-cloud solutions are eventually going to move to the cloud, or will there always be a significantly large non-cloud solution set out there? There's also the question of how many of the cloud customers would leverage the providers' default tech stack choices versus customers making their own choices.

Lots of cloud migration/workload tools out there - OneOps (by WalmartLabs) is one example. Check out http://oneops.github.io/admin/getting-started/#installing-va...

My main concern is the security angle, as the "Big 3" are all pushing for a barrier across country lines. I'd prefer any of the data I store in the cloud to stay in a US cloud.

I wonder how much a slowdown in VC investment would hurt their sales. They are a great operating unit, I really wish they would break it apart from Amazon major.

It depends on what level VC we're talking, and what % of their IaaP service is purchased by said service.

There might be a hiccup in earnings if the tech sector crashes.

The article is interesting, but it fails to consider one aspect. AWS is a line of products, very broadly defined, instead of a single product, with different level of maturity. You go from services like EC2 that are IAAS to RDS and co. that are PAAS to some foray into SAAS.

I don't have first-hand insights, but my guess is that Amazon is working on many more products and offerings. And if it's not Amazon, are company running on AWS.

So, I'm pretty sure that having a break-down of the internal numbers between service we could see many different growth curves superimposed. Some that still experience exponential growth, other still grow but with a slow-down. The sum is still an exponential curve.

tl;dr: The growth will profile will depend a lot on the pace of product innovation AWS is able to provide. And that's different from, for instance, the Amazon Marketplace where the only innovation is going after a new category.

As a consumer of AWS's services, every time I start to think "I could really use this service..." or I have a "If AWS had this..." thought, they seem to release something that addresses this concern. If they added multicast within VPC I would be the happiest lil' AWS camper.

Reply on news.ycombinator.com