S3Proxy allows applications using the S3 API to access other object stores | Hacker News Reader

S3Proxy allows applications using the S3 API to access other object stores

github.com

47 pointsgaul16 comments

I was considering something like this to have a local (or local networked) cache store for S3, I am doing 99,9% of readings, so it could be nice to put the cache in the S3-layer.

Does anyone knows if that exists?

I think you can setup nginx proxy with cache. Unfortunately, that's easy only for public files and authenticated requests won't save you much (for that you can also write a small script to authenticate yourself if there's a cache hit).

Oh, yes, thanks for the suggestion. I will see if I can use NGINX as a proxy, all my files are public anyway.

http://www.mccarroll.net/snippets/s3boto/index.html

https://code.google.com/p/bigcache/

How does this work when the object stores don't support all the features of S3?

For instance Atmos doesn't have public buckets, DNS names for buckets, named keys implemented like S3, regions, etc.

But it does have features that S3 doesn't have like byte range updates, erasure coding, etc.

S3Proxy cannot provide more that the S3 API allows, such as byte range updates, although some features can be enabled by bucket-level policies. S3Proxy can emulate features that other providers lack like public buckets and could provide other features like bucket-in-hostname. Apache jclouds provides the underlying object store compatibility so S3Proxy will benefit from improvements to it.

Little help, please.

We have a process that uses s3express (aka using the S3 API) to copy files from local filesystem to S3.

S3Proxy would allow the process to copy files to [another_server, atmos, azure] ?

Yes.

This is great timing for me, currently working on trying to iron out some bugs in a project with S3, and this helps me eliminate one line of inquiry. Nice project!

Very cool project. What sort of latency does the interface add? I imagine not much, but I didn't see any stats in the readme.

Agreed. Even I was looking for the statistics in the Readme. May be, they will add after seeing your comment.

Surely the latency cost would be trivial compared to the inherent latency of using an off-site storage anyway?

I'm not sure why you're both asking for it, bit strange given how tiny the cost of using a generic interface is compared to the cost of sending a file over the internet.

Shouldn't this be solved on the client level via proper adapter support?

I like how it can be attached to a black box though.

[the author of s3proxy is my friend]

It seems silly for every client to implement support for every single object store out there. Libraries like jcloud can handle it, but then it's tricky if your client is not written in Java.

Also, sometimes it's not trivial to add support to existing projects. For example, you can use s3fs with s3proxy to store your files on non-s3 object stores.

Does this work with the s3cmd CLI?

Unfortunately s3cmd requires bucket-in-hostname support which S3Proxy lacks:

https://github.com/s3tools/s3cmd/issues/149

https://github.com/andrewgaul/s3proxy/issues/8

I also added a compatibility list wiki:

https://github.com/andrewgaul/s3proxy/wiki/S3Proxy-compatibi...

Reply on news.ycombinator.com