MimicDB: Speed boost for S3, 2000x faster | Hacker News Reader

MimicDB: Speed boost for S3, 2000x faster

mimicdb.com

21 pointsnathancahill14 comments

This is pretty cool. How would you start using MimicDB with an existing S3 bucket?

It sounds like it provides a nice s3 cache system for your backend. That is, it matches every s3 call to be a simple drop in request replacement powered by redis for caching.

Disclaimer: I'm tired and may have read it wrong.

It says that it only stores metadata locally not objects themselves (although it could probably have an option for that too and serve as a general S3 cache).

Nice effort, Redis is perfectly fine but I believe that the storage layer should be somehow more separated in case someone wants another type of storage, e.g. in-memory SQLite is adequate and already installed in most systems.

I agree about with you about a replacement for redis. Maybe it's only me, but I seem to be relying too much on redis for simple things. I feel that soon enough I'll need 4 independent redis servers for my production environment.

That's of course tongue-in-cheek, but there's a degree of seriousness.

Good idea. Maybe in the future it will support different backend databases. For now, Redis is perfect.

Yes, I think that's the gist of it. It looks like it assumes it has seen every call to the bucket it's working on though. So to start using it with an existing bucket, you'd have to warm up the Redis cache by downloading all metadata from the bucket?

It looks like a drop-in replacement for boto's S3 interface, so if you're already using that, you can probably just change the imports to refer to MimicDB.

I think it is a proxy for S3 that sits between your application and S3. So it works with Boto, or your favorite S3 library.

Exactly. If you're using boto.s3.connection or boto.s3.bucket you can change that to mimicdb.s3.connection and mimicdb.s3.bucket.

You can call the .sync() function on an S3Connection or a Bucket object. The MimicDB instance is populated with the current data on S3. Also, syncing is stateless, so you can call it anytime to insure consistency.

Would this work (in the usual cases) where you have more than one server interacting with S3 ?

Probably you can, if you implement upload callbacks which can be passed through the success_action_redirect parameter. Not sure though.

Yes, absolutely. You can connect all of the servers to the same MimicDB instance as you would with Redis.

I think you mean "isomorphic" not "isometric". Neat project.

Reply on news.ycombinator.com