Show HN: Amazon S3, Microsoft Azure, Rackspace, Backblaze via a Single API | Hacker News Reader

Show HN: Amazon S3, Microsoft Azure, Rackspace, Backblaze via a Single API

cloudrail.com

191 pointstmm_65 comments

Do all these services provide equivalent or abstractable guarantees? Amazon S3, for example, provides 'read-after-write' consistency, meaning once you've received a positive response to a put operation, you can expect to be immediately able to retrieve that object. But it used to be 'eventually consistent', meaning it was possible to receive a positive response to a put, but then not be able to read the object immediately after.

Similar guarantees are needed around how soon after deleting something can readers expect to get 404s.

If these guarantees differ, you might find abstracting over the stores doesn't work the way you'd like...

Very good question. Of course all services behave a little differently which is out of our control. To be honest, I don't have an exact answer on that. In my first tests I could chain a download directly after the upload with all services. But I don't know if that's because they're all guaranteed to be read-after-write or some were eventually-consistent just really fast. In general, if one service is read-after-write and you want to switch it for one that isn't you might get problems, unless you've programmed defensively (checking for existence before proceeding). Give us some time to check that and run some more tests.

That's kind of the point though. This abstraction seems doomed to be extremely leaky. Honestly, I'd rather just deal with them separately than be constantly fighting an API.

S3 only has read-after-write consistency for PUTs to new objects (not overwrites), and if you do a HEAD/GET before the PUT that degrades into being eventually consistent.

http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction....

It's so bad that big companies, like Netflix, have to write a whole service to store a copy of the s3 metadata: http://techblog.netflix.com/2014/01/s3mper-consistency-in-cl...

Netflix have turned the S3 metadata eventually consistent problem into s3mpr metadata eventually consistent problem. The difference is that they can now inspect and reason about s3mpr's metadata.

Spotify have had to do the same thing for Google Cloud Engine. I can't help wonder if eventually consistent object stores will go the way of NoSQL databases, when a consistent, scalable hierarchical filesystem appears.

>I can't help wonder if eventually consistent object stores will go the way of NoSQL databases, when a consistent, scalable hierarchical filesystem appears.

Of course they will. Eventual consistency is a huge tradeoff that I don't think anyone would make if they weren't forced to.

Maybe something like this - 1 million ops/sec on HDFS: http://www.logicalclocks.com/index.php/2016/10/14/hops-smash...

Then you trade availability with consistency. No system can't escape CAP

Exactly. Availability and partition tolerance are often hard requirements, so we have to do all sorts of gymnastics to deal with eventual consistency. In personal projects where availability isn't a big concern and my most precious resource is my own time I tend to make different tradeoffs.

GCS is a bit better, since individual objects are strongly consistent, but I've been bitten by the eventually consistent listing.

Also on GCS, if you do a HEAD after DELETE on a bucket that is under lifecycle management it returns 200 instead of 404. Not really a consistency issue but it can really come and bite if you if you're not aware of it. GET returns 404 but HEAD returns 200.

I reported it as a bug but Google said it was by design. More specifically they said: "You are correct, if the versioning enabled in your bucket then the object metadata is saved as an archive object in the bucket [1].This is the reason you are getting 200 for your HEAD request."

You can compare these guarantees and results from testing:

https://github.com/andrewgaul/are-we-consistent-yet

Thanks! We'll address this in our documentation.

That is pretty old, considering what S3 has changed in the past two years. In particular, it talks about reading S3's us-standard region from the East and West coasts, and that has evolved to where us-standard (which is really us-east-1) is now the same as all other regions when it comes to consistency (presumably because all requests physically go to the East coast now, but I haven't tested that):

http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction....

Fair point, I will update the AWS us-standard results and include Backblaze as well.

Per that AWS page, you might want to add an asterisk that you lose read-after-write if you make a GET or HEAD to the keyname before your first PUT. So checking if the object exists will punk you down to eventual consistency.

Open source alternative: Rclone -- http://rclone.org/

I switched to Rclone recently for personal backups and it's unbelievable how slick, stable, and well-behaved this program is.

As far as I can tell though, it does not expose an API beyond its command line interface though.

It's open source golang, just copy and paste if you need to. Or send a PR for an api.

Wish I knew about this a week ago..

I'm using CrashPlan to backup from my laptop to another computer on the network. I see that rclone can backup to a local filesystem. Do you happen to know if I can I do with rclone what I do with CrashPlan? i.e. use rclone to backup my local filesystem to a remote server?

It looks very sleek though, I'll have to give it a try.

This is our brand new unified API for enterprise cloud storage providers. It's part of the CloudRail API Integration Solution which consists of multiple unified APIs for different categories like social, payment, consumer cloud storage etc. Our value props are: A single API for multiple providers & No API changes since we keep the integrations up-to-date. All that without a hosted middleware. So we never touch the data. Looking forward to hear your feedback.

Looks like a handy set of tools. Check the spelling of 'Download' on the linked page :)

Thanks. I'll correct that typo asap but currently I don't want to clear the cache :)

Why not allow people to specify URLs ? e.g. Swift (what RackSpace runs) can be run anywhere, Ceph can imitate s3 or swift, and Eucalyptus can allow people to run API compatible versions of AWS.

Providers can also add new regions at the drop of a hat.

Limiting it to pre-defined URLs is a pain, and relies on tools updating for new regions being added.

S3Proxy offers similar multi-protocol access at the S3 API layer:

https://github.com/andrewgaul/s3proxy

It has compatibility with local filesystem, Backblaze B2, Google Cloud Storage, Microsoft Azure, and OpenStack Swift. Note that this is software and not a service.

[S3Proxy author plug]

is this compatible with django-storages (which supports S3)

S3Proxy uses Apache jclouds underneath which has broad compatibility with object stores including S3 clones. However no two S3 implementations are alike so you will need to test.

>Note that this is software and not a service.

It appears that Cloudrail not a service either, just a commercially licensed library (like Qt for example). In fact, I find their simple library approach a bit less awkward than your local server approach, though I suppose that was necessary in order to make a neat drop-in replacement for code that's already targeting S3.

The Rackspace API is OpenStack Swift, a project used by many more public and private cloud providers than just Rackspace.

I've been working on Swift for the past 6 years, and I'd be happy to help with your OpenStack Swift API integration. See my profile for contact info.

Sounds amazing. I'll make sure to reach out to you tomorrow.

Also check out libcloud by Apache. Pretty solid Python API wrapping various cloud services. :)

Definitely a good alternative until we have a Python SDK :)

No monthly fees, support open source: http://jclouds.apache.org/

Wouldn't surprise me if that's what they're using behind the SaaS

Nope, we don't use it but it is a very cool project. We have SDKs for Android, Objective-C, Swift, Java and Node.js so far. That's why we couldn't us it anyways. We don't want to compete with open source projects like this. Our goal is to offer easy integrations for all platforms and all major use cases (not only cloud storage). Btw, to support open source, we made our solution free for non commercial projects.

Is anyone going to do one of these in C#?

Also libcloud https://libcloud.apache.org/

It's a nice feature-set, looks well-designed, and handy - but anyone using one of these APIs would find it super trivial to implement this themselves? I'm not sure I see the cost-benefit making sense here? I'm keen to be convinced, though - what's the killer use case? (I'm not aware of APIs changing or being deprecated in any big way)

Edit: ah, the other APIs make way more sense to me. You can offer multiple options to a customer. But this doesn't seem to have an equivalent?

This allows you to easily integrate multiple of these providers if necessary or easily switch. Of course you can integrate them one by one on your own but that takes a lot of time time. With CloudRail it's one API which works even cross platform. Our ultimate goal is to handle all your integrations and not only cloud storage. So unified APIs for fast API integrations and API Change Management to keep your integrations running forever.

so how exactly would changes in API be handled?

Our system monitors the APIs and informs us about changes or the provider does. This happens usually months before the integration would actually break. Afterwards CloudRail updates the SDKs and informs the affected users via email and in the portal. And affected means really affected, so only if you use this specific (broken) function. All you need to do then is update the SDK to it's latest version. We are also working on a optional and completely automated way to update the SDK. But most developers want to test it before. Btw, any opinions on the auto update here?

I see what the feature is, but I don't see the use case or benefit - in this particular case! The social, payment, consumer storage, I definitely do. But it feels a little like the age-old example of abstracting away your database layer, in case you wanted to change which database you use later. (It's incredibly rare to do it!)

Even if you don't want to switch and believe in non changing APIs, it is easier to integrate with CloudRail :) But as already mentioned, this interface is really part of a bigger offering. We want to handle all your integrations eventually and enterprise cloud storage has to be part of that.

Can you provide a use-case for when you would need to be able to easily switch (or potentially hot-swap) cloud storage? It sounds like some of the justification for ORM's; while it would be nice to have if it took no more time to integrate with that API its hard for me to see how it could justify extra effort in implementation/maintenance unless you have a concrete understanding of when you need it.

Pricing.

It's (kind of) like Segment for backend storage services.

Its nice but what incentive does cloudrail have to provide perpetual gratis access to someone elses API? Unlike an app that can still work when the original developer goes broke, an API requires that always on access.

Does my whole build break because cloudrail can't handle the traffic anymore? At least with AWS or GCS or Azure, etc, one has a business relationship with the API provider and so can keep the lights on.

CloudRail doesn't touch the data. Everything flows P2P between the app and the provider. It's like if you integrate the native AWS SDK. So even if CloudRail goes completely down, your integrations will keep running.

Happy to see you guys having success! I've done my master at the University of Mannheim. Never thought that this would happen, but I kind of miss the city.

Thanks. Viele Grüße aus Mannheim!

Viele Grüße aus Passau! :)

Stuff like this isn't exactly new. You get lowest common denominator functionality.

Simon Wardley summed this stuff up pretty well about 6 years ago, http://blog.gardeviance.org/2010/08/islands-in-sky.html

please add "storage APIs" to the headline. without it, it is misleading

I know, but it's the limitation on characters for headlines on HN. Isn't it clear with S3, which is storage only?

I was looking for the same thing but for email providers. I would like to let users to subscribe to mailchimp,aweber,getresponse etc using a single integrations instead of having to integrate each one of them.

We have a very simple interface for email here: https://cloudrail.com/unified-email-api/ But it just offers sending emails. What you described is actually a potential candidate for a next interface. Would love to discuss that use case with you. Feel free to reach out: support@cloudrail.com

No Google Cloud Storage? It even has an S3 API you could use...

Yeah, seriously speaking. Azure, Rackspace and Backblaze before Google? lol.

Isn't this what Hashicorp's Terraform also provides?

When I saw this on HN... popped in here to say this exactly. I'm interested to see what folks have to say?

As I understood it, it is more a solution for dev-ops to create the infrastructure. Like a provider agnostic CloudFormation. CloudRail is about making it super simple to integrate APIs into an app.

Not quite.

Terraform abstracts the APIs of multiple cloud providers into HCL, HashiCorp Configuration Langauge. What Terraform is essentially doing is enabling you to skip the part where you write code to talk to AWS' API, allowing you to spin an EC2 instance, or talk to GCE and store something in GCS, and instead just work in one language, via one tool. If the underlaying APIs to those providers shift and change over time, HashiCorp updates the logic under the hood within Terraform, and your code continues working. What it does not do is provide a single resource type or function/method that you can use to upload objects to both AWS and GCE - you have to write two separate resources to work with each of the two example cloud providers.

What CloudRail is saying is: the upload() function will work whether you tell it to push the object to S3 or GCS.

What Terraform forces you to do is change `resource "aws_s3_bucket_object" "picture" {}` to `resource "google_storage_bucket_object" "picture" {}` when you want to change from uploading to GCE from AWS.

Make sense? :)

Is this not simply a library that unifies multiple services into a consistent abstraction?

Your Android example has strange quotes on the .createBucket(...) call -- line 9

There's a typo: "This means for example the function donwload()"

Thanks! Will fix that soon.

Reply on news.ycombinator.com