Hacker News Reader

Comment by nathancahill | Hacker News Reader

Also see MimicDB [0]. Runs a transparent cache that responds to most S3 API calls locally using Redis to store metadata. Besides the cost savings, it's extremely fast. Listing an entire bucket with millions of objects is close to instantaneous.

[0] http://mimicdb.com/

View on HN

It also adds another piece of infrastructure that needs to be maintained and can go down. Not necessarily the best option for everyone.

While the caching would be more effective with one large centralized instance, I think the intended use case is to have one cache per server. So then it's not really extra infrastructure.

Actually, you can do it either way since it's backed by Redis. You can set all servers to connect to the same Redis instance, or run them all individually.

Do you use Redis clustering in this case (otherwise how would the cache stay consistent)?

What if Redis goes down?

"What if X goes down?" is my new favorite straw man on HN.

Regardless, it's just a caching layer, and requests are passed through to the API in that case.

There's always a single point of failure.

Is their a SaaS version of mimicdb? I'm ready to signup right now.

No, and it wouldn't help much if there was. The biggest benefits come from running the metadata cache locally, or at least on the same local network. The cost/time savings come from preventing external API calls.

If I have a bucket with 2 million key, it takes a long time and 2000 requests to get the dir listing. With mimicdb it would only be one fast request.

That's true if it was local. A 2 million key response from an externally hosted MimicDB instance would probably have to be split in 2000 requests and take the same amount of time.

Send me an email nathan@nathancahill.com and I can show you how to set up a local cache. It's quite simple.

Reply on news.ycombinator.com