A peek at the massive scale of AWS
enterprisetech.com
enterprisetech.com
As mentioned in the slides, that's the same latency usually associated with SSDs and 100X better than the latency typically associated with cross-region networking. This suggests to me that running distributed databases in multiple AZ's has almost no latency penalty (e.g. you won't even paying eventual consistency / replication lag taxes that you might think are a danger). That's pretty damn cool.
The big takeaway from Hamilton's talk was actually something that dawned on me when they released the c3/m3/i2 instances which is what they have done with SR-IOV.
The software stack that implements the AWS VPC fast path must be running on the NIC itself. This is a big deal. Being able to implement the require isolation/tunneling/encapsulation logic and routing in the NIC is a huge performance boost and drastically simplifies the hypervisor. Well, the software portion of the hypervisor atleast.
If you listened closely to the talk he laid out the differences in latency impact of each layer in the networking stack, from the fibre (nanoseconds) up to the software stack at some number of milliseconds, several orders of magnitude worse than anything else in the stack.
The reason why the quoted latency is so large is how para-virtualized networking is implemented. In order for it to be (bandwith) performant it needs to use large ring buffers with many segments. This is very throughput optimised and latency suffers a ton as a result.
By moving all of the queuing to the NIC you get a bunch of benefits, namely the dom0 (basically the host in Xen terminology) is no longer involved in pushing packets and you are not incuring the cost of the Linux networking stack 2x. In the paravirt model the skbs are transferred across the circular buffer into the host OS where they are injected into a virtual interface, thus traversing the full net stack again.
In the SR-IOV model the address space of the virtual NIC is mapped into the guest OS using Intel's IO-MMU extensions and the guest is then able to communicate directly with the NIC, thus 100% bare metal performance.
If SR-IOV was the only improvement it would be impressive, however it's the consequence of it's existence which makes the biggest difference. If the guest is talking directly to the NIC, then all of the encap/decap is HW accelerated too and in theory this means the full networking stack is end-to-end in HW.
MPLS has a bunch of advantages over VXLAN and is decidedly L3, which is more suited to AWS VPC. The real question however is what the control plane looks like.
They may just go pure carrier style and just do tagging and resource control on the NIC but it's entirely possible for the NIC to be doing much more, especially if they have a dynamic control plane similar to OpenFlow (though probably not exactly OpenFlow.. it's less useful than it sounds).
I would love to see a talk on AWS VPC internals rather than sitting around taking pot shots. :)
Yes, Hamilton points that out in his talk: https://www.youtube.com/watch?v=JIQETrFC_SQ
2 ms for, say, 3 km is 1500 km/s, or around 0.75% the speed of light in fiber (~200,000 km/s).
I wonder what the bottleneck is.
On a much much much smaller scale, this is also what I'm working on during my current sabbatical: https://ripe69.ripe.net/archives/video/177/
I thought it may be relevant to post, as we our code is under GPL and willing to collaborate with anyone.
This the first reference I've heard to scales of economy and "Blast Radius" concerns (I.E. How much damage occurs if a data center goes down - apparently Amazon feels that at around 80,000 (or so) servers, it makes more sense to build new data centers, than to make existing ones bigger.
This is why Availability zones have multiple data centers (as many as 6 (10?) in US-East)
Also, while I was aware that Amazon was looking at building their own network stack - I wasn't aware that they'd replaced all their Cisco/Juniper gear with white-label ODMs with their own custom software stack. Now that's a company that takes networking seriously.
I decided first to try Google - specifically Google App Engine. Just to see if I could get a plain vanilla base case working quickly. And my initial reaction was that AWS is still head-and-shoulders above everyone (or at least Google). The Google UI and setup process seemed ridiculously complicated and unfriendly. With AWS, I was up and running almost immediately. Not so with Google.
So I immediately ran back to AWS and dropped Google. I'm not sure if my bad experience with Google was because I had framed my expectations through my AWS experience and thus wasn't able to use Google the way it intended to be used. But it just seemed way too unfriendly. It seemed to require needless dev installs that should just be automated.
When I went back to AWS, just the sheer amount of services they offer seemed staggering by comparison. Google still has products in BETA that AWS has in mature mode.
I was going to try out other competitors like Azure, Digital Ocean, etc. but now feel like there's no need. AWS is just good.
AWS is a high-quality, extensive offering, but it's not suitable for every situation. The 'sheer amount of services' are in some cases lacklustre reimplementations of services you could run yourself on EC2, for example.
What this boiled down to is "I wanted to compare things to AWS, so I had a half-assed look at something that really isn't a competitor then immediately stopped looking." That's not really convincing.
A perfect example would be the difference between GCE networking and AWS VPC. GCE networks support routes with priorities and if you insert multiple routes into the routing table with equal priorities it does what you would expect, equal cost multi-pathing.
This makes it really easy to implement proper scalable NAT for private instances, which is just pure pain in AWS VPC.
There are many more examples of this and AWS is not the only culprit, both GCE and Azure have either missing features or mis-features that make me want to flip a desk sometimes.
As cloud users and developers, we should be pushing for standardization between vendors. I dislike having a huge ecosystem of proprietary cloud products to learn before I can do anything useful with them.
I try to stick to IaaS offerings because it's easier to evade vendor lock-in that way. I'm happy to see Docker taking off, because it gives us a useful abstraction and platform that we can take with us across cloud vendors. (And isn't it interesting that AWS is rolling out their own Docker management platform? I think that's a good thing, but it's also true that AWS has an incentive to compete with more open systems like Kubernetes rather than adopt one.)
Basically nothing beats AWS on IaaS and Azure seems to be the best for PaaS. I'm comparing in term of cost, feature and ease of implementation/maintainability.
But the surprising thing, even to Hamilton, was that network availability went up, not down.
Networking, and storage (to a lesser extent) seem to have enterpriseits. By that, I mean the companies involved in it do everything in their power to maintain their high margins. One of the ways they do this is adding features that everyone just has to have (or so sales will tell you). Even when many of those features are things that should _NEVER_ be actually used in practice, due to their affects on performance or reliability (take WAN accelerators for example, otherwise known as how to make your network appear faster 90% of the time and completely non functional for the remaining).
What I really wish is that the amazon's, microsofts, googles, facebooks, etc would get together and actually release these switches/etc they are building. While I imagine they never will because they view it as a competitive advantage. It sure would be nice to have some of this stuff available for medium sized data centers without having to spend $$$$$ with the established vendors just to get something that can do 100Gbit.
A design for network switch hardware and software was recently announced: https://code.facebook.com/posts/681382905244727/introducing-...
http://wikimapia.org/#lang=en&lat=53.293772&lon=-6.349669&z=... http://www.punchconsulting.com/our-projects/logistics/tesco-...
Besides, I don't remember examples of Amazon being nice - not that it doesn't happen but they don't communicate on it.
Not really true. As part of an annual capacity planning exercise each team was required to plan and scale for holiday peaks. Infrastructure is not "free" and each team has to optimize for good performance at an affordable cost.
> All that spare capacity became AWS.
Not true, never was. I regularly reviewed and provided feedback on the original narrative document for AWS. While I don't have a copy handy, I am absolutely certain that the document was focused on providing infrastructure services to developers.
> I bet their utilization rates aren't any different today ...quite possibly worse.
I don't have access to those numbers, and have no permission to share them even if I did. Your thought model for utilization needs to take the EC2 Spot Market in to account. Savvy users of EC2 have learned to optimize their large-scale compute jobs and their bidding process to gain access to what would otherwise be (to your point) underutilized capacity.
The recent "Gojira" run by Cycle Computing (details at http://www.marketwired.com/press-release/cycle-computing-sof... ) is a great example of how the Spot Market can be used to great advantage by clever developers.
The way I think about it is - can utilization rate grow at the same rate or higher than machine count at their data centers and for how long?
With all the levels of virtualization available and their market leadership today that curve can look quite magical I accept. But for how long? Seems quite a shaky curve to be betting 5 million machines on.
The excess capacity story is a myth. It was never a matter
of selling excess capacity, actually within 2 months after
launch AWS would have already burned through the excess
Amazon.com capacity. Amazon Web Services was always
considered a business by itself, with the expectation that
it could even grow as big as the Amazon.com retail operation.