Hacker News Reader

Comment by TheDong | Hacker News Reader

Many amazon servers can't service requests from a browser (due to CORS headers), and all of them require an aws account.

Providing an AWS account that people could use is a security nightmare, and many of the operations have charges associated with them.

If I provide my aws credentials, then he could proxy through his server and request on my behalf, but I wouldn't want to give him my credentials and I might as well just play with the aws-cli which exposes pretty much all of the same calls anyways.

View on HN

Sounds like a cool project to blog about: in-memory implementation of all AWS APIs.

I was considering writing something similar: a "semi-mocked" implementation of all of AWS, getting you most of what you get from running OpenStack but from one static binary running in userland and with no persistence.

My use-case for it was that I'd write it all in Erlang, and then each public port on a mocked EC2 instance would have a single Erlang process hooked up to it to respond to messages coming to that instance-IP:port. So you could set up something that looked like a thousand-node distributed VPC, from the perspective of an AWS client talking to the API, and it would all sit in a couple MB of RAM and near-zero CPU.

This! A local mock of AWS is sorely lacking for most services. The best one I know are Amazon's own DynamoDB and the independent FakeS3 tool which exposes an http api and saves the blobs and metadata as local files in a transparent way.

I've been waiting for someone to come along and say "I did this!"

Okay so I didn't do this, but I have seen several folks where I work create some stubbed AWS services to facilitate testing. I'll have to ask them if this turned out to be worth the (gargantuan) effort.

Edit: Looks like there are a couple[1] instances[2] where people have done this to some degree. Many of them are very old, though. The one built by a team where I work only implements SNS/SQS as well, so there's a pretty big gap for other interfaces. It seems like the recommended way to do this is to just use something like EasyMock or whatever and stub rather than have an actual API implemented for integration testing. I'm not sure I'm satisfied with that answer; on the other hand, it's substantially easy for a custom implementation to wildly differ from the actual AWS API behavior... Seems risky to base integration testing results on that?

[1] https://github.com/p4tin/goaws

[2] https://github.com/unbounce/yopa

Here's some third-party implementations of some AWS APIs, with no endorsement of quality:

[A1] https://github.com/jubos/fake-s3

[A2] https://github.com/scality/s3

[A3] http://s3ninja.net/

[A4] https://github.com/basho/riak_cs

[A5] https://www.npmjs.com/package/s3rver

[A6] http://ceph.com/ceph-storage/object-storage/

----

SQS / SNS

[B1] https://github.com/iain/fake_sqs

[B2] https://github.com/yourkarma/fake_sns

[B3] https://github.com/adamw/elasticmq

[B4] https://github.com/p4tin/goaws

[B5] http://stratosphere.codeplex.com/

[B6] https://github.com/unbounce/yopa

----

DynamoDB

[C1] https://github.com/mhart/dynalite

[C2] https://github.com/ananthakumaran/fake_dynamo

----

SimpleDB

[D1] https://github.com/stephenh/fakesdb

I can't speak for all the libraries linked above, but I've used fake-s3 with a rails app and s3rver with a node app. They both worked great for development and testing. I'd definitely use em again.

I think it's worth noting that some of these libraries have different intended goals. For example: Riak CS is intended as an alternative that you can use in production, while s3rver is meant for testing.

Another project to be added to this list would be

https://github.com/minio/minio - Minio is an object storage server compatible with Amazon S3 and licensed under Apache 2.0 License.

Disclaimer: I work for Minio.

+1 on just using the CLI but could also use STS to generate a token with a locked down policy client side and the risk of proxying would be much smaller

"Many amazon servers can't service requests from a browser (due to CORS headers), and all of them require an aws account."

The first part is rapidly becoming less true; any of the services enabled by default on this page support browser calls by default.

https://sdk.amazonaws.com/builder/js/#

There was a similar HN post like this but for Elasticsearch a few weeks ago.

I'm kind of thinking that a command line tool for elastic search similar to the AWS command line isn't a bad idea. In general when I can't remember an AWS API call, instead of googling, I just use the AWS cli. I find it faster than google or a reference like this. Anyone else?

Reply on news.ycombinator.com