Libcloud: One Interface To Rule Them All

  <link rel="stylesheet" href="/styles.css">

  Link: </styles.css>; rel="stylesheet"

1. http://tools.ietf.org/html/rfc5988

EDIT: hm. No idea why I got downvoted, but I tightened it up to get more to the OP's point.

There's a great Python library called UniversalClient which illustrates this nicely: it provides some helpful scaffolding for working with REST APIs and structuring your code in a maintainable manner but ultimately a programmer still has to provide the service-specific understanding:

https://universal-client.readthedocs.org/en/latest/basics.ht...

I don't get how that would work magically. Imagine that I want to update my phone number in my account data on two different sites. If I point this magical library at the two site's APIs, how would it decide which resources to update under which paths?

It seems to me that you would still need to implement the different concepts, resource names, and so on across different sites, and you wouldn't really win anything here.

All drivers available in Apache Libcloud have been implemented without introducing a dependency on a custom SDK. Why make an exception for Azure? Note that the complexity does not come from the Azure Python client API but rather from the Azure REST API that is in my opinion much more complicated that any other provider's.

Azure has some specific idiosyncrasies (such as deployments, certificates management, VM images that first need to be transferred to a blob container) that make it harder to fit in the simple and unified Libcloud API.

[1] https://github.com/WindowsAzure/azure-sdk-for-python

[2] https://github.com/WindowsAzure/azure-sdk-for-python/blob/ma...

[3] https://github.com/WindowsAzure/azure-sdk-for-python/blob/ma...

But there's no stalemate in OpenStack: it supports both APIs, although the EC2 API definitely gets less love. If you improve the EC2 API implementation, those improvements will get merged subject to the normal code review. If you want to do an ecosystem project that proxies EC2 calls to OpenStack, there are obvious questions over code duplication, but there's not profound opposition. And if you do want to do a proxy, go right ahead; you don't need OpenStack's buy in at all: build it, make it better than what is there already, and you'll win the argument.

The root problem (as with all open source) is that there are are a lot more people saying they want others to build better EC2 compatability, than there are people saying they want to build it.

Eucalyptus has no more or less 'embraced' the AWS API methods than OpenStack has - save for a bare few methods that have different connotations in OpenStack land than they do with Euca: https://wiki.openstack.org/wiki/Nova/APIFeatureComparison.

The 'political battles' you are citing in OpenStack are actually a bunch of cage rattlers who are promoting their own interests in their respective service offerings. Going into a company that uses AWS for infrastructure and trying to pitch them on OpenStack is a long, drawn out process. Selling new technology is hard.

The lowest common denominator benefit that OpenStack provides is an open and transparent code base that can be modified and utilized by anyone. It's that openness that allows the few people using features like VPC and autoscaling to write some simple code to do deployments on whatever they like. Honestly, it's a handful of lines of code at the most for any given method. Saying you 'lose' feature functionality is simply a rationalization for attacking the OpenStack movement.

That's not the level where libcloud is aiming for LCD - it's not at a provider level, but at a product level. As long as a couple of providers support autoscaling, and AWS and Rackspace do, there can be support for it. It's on my todo list to implement, by finding the common thread between AWS and Rackspace APIs, and developing an abstraction that aims to cover the features of both.

[Rackspace employee, libcloud committer]

The AWS API is convoluted, overwhelmingly complicated, and often yields no insight to the actual behavior taking place in the Amazon system. I can think of countless scenarios wherein my API call succeeded but nothing happened, with no reporting of the reason why anywhere. A later run of the exact same call again works. No explanation.

Now, arguably, I'm discussing the implementation of the specific API, but even on an architecture level the AWS API tries to handle absolutely every case in the most complex manner possible. This is because AWS tries to handle every case in their systems; where other providers trim things for simplicity, AWS adds as much as it can. This flexibility can be useful if you need it but to someone getting started, it is a giant wall. They've gotten much better about guiding people through setup, but the API still reflects the kitchen sink philosophy. Everybody I've ever known who loves the AWS API has only ever used boto. As an exercise, launch an instance with ephemeral and EBS storage mounted using the bare API on a clean account. You'll see what boto hides from you.

The correct approach is something like libcloud with the ability to interact with platform-specific things in a way that the library does not anticipate. However, there's a school of thought in that if you are digging yourself in to platform-specific features, your ability to move is now a significant line item of technical debt that could put you in a position where you are completely stuck and cannot hire enough people to get out.

As a former hosting employee it is immeasurably frustrating that everybody mentally defaults to AWS these days without even a second thought. All of Silicon Valley is basically wired to funnel venture capital directly to Amazon. The VC firms might as well just invest in Amazon and skip the startup risk, since inevitably most of the cash burn of a startup will go directly to AWS (because it takes a genius of reservations, flexible scaling, and other tricks to not spend six figures a month on your hosting). AWS was designed for flexible scaling, not running your 700 instances 24x7 regardless of load, but nobody operates that way. You think Amazon is going to tell you that?

With Amazon, the API is full of issues, and that's even before speaking about the service itself as well as the support; even with paid support, 12-hour ticket responses were the norm for me including in an outage situation. My personal favorite was a six-month ticket that culminated in "we don't know, so we're closing this ticket," when reporting buggy behavior with their API. There are many aspects of AWS that are dreadful, the API being only a small example, and the suggestion that we should standardize on the AWS way of doing things as an industry is very harmful. I'd contend we need a viable alternative, but Amazon has made themselves a Kitchen Sink Provider and other companies can only bite at little pieces of the pie.