InfiniCache: In-memory cache that is built atop ephemeral serverless functions | Hacker News Reader

InfiniCache: In-memory cache that is built atop ephemeral serverless functions

github.com

64 pointsmastabadtomm23 comments

If distributed, lazy, just-in-time, global cache is what you want, Cloudflare throws it in free (with a few limitations) with Workers [0]. It is so much more straight forward than InfiniCache. I use Workers to cache content from few KBs to high MBs sourced from single-region S3 (could use Scaleway, Backblaze, or Wasabi for even more savings [1]) and DynamoDB (Workers KV can store 25MB [2] and is a viable alternative, but then cache may not be even required).

[0] https://developers.cloudflare.com/workers/runtime-apis/cache

[1] https://www.cloudflare.com/bandwidth-alliance/

[2] https://developers.cloudflare.com/workers/platform/limits#kv...

Interesting design, but it seems viable only by exploiting "free" cacheing given to AWS lambda functions. There are no guarantees here from Amazon, and I'm pretty sure whatever free capacity is there right now would evaporate once people started to exploit it more. AWS lambda should be more convenient than manual scaling, but never cheaper than hand-optimized service implementation.

For serverless to be viable, especially for languages like Java, instance reuse is really important, otherwise every call results in a cold start with unacceptable delays. So serverless provides can't really do much here.

They could rise prices, but that will likely hinder adoption.

I suspect they could do a fair bit. An obvious thing is that they look for this pattern of behavior and turn up the speed with which they kill processes. They're still meeting their terms, and the normal Java users never notice.

A more subtle thing is just to tune the lambda eviction code to be aware of profitability. The instances that get evicted first are the least profitable, the ones that have the worst ratio of billable to unbillable GB-miliseconds. Good customers get good performance. And the other ones say, "Gosh, cold starts are expensive; I'd better pay extra for provisioned concurrency."

I've been toying with the idea of using some form of memory suspension. The theoretic cost of a cold start them becomes just the time it takes to rematerialize the memory from disk. Has anyone seen anything similar out there that could be a starting point?

Cloud Run could be what you are looking for (I'm not affiliated with it, just a regular costumer).

https://cloud.google.com/blog/products/serverless/cloud-run-...

Name reminds me of horrors of JBoss Infinispan cache

I actually thought it was a reboot of Infinispan cache at first

It's not? Bummer.

Infinipain

Very important caveats from the paper review at https://mikhail.io/2020/03/infinicache-distributed-cache-on-...:

> Mind that the workload was selected to be a good match for InfiniCache properties (large objects with infrequent access).

> The hourly cost increases monotonically with the access rate, and eventually overshoots ElastiCache when the access rate exceeds 312 K requests per hour (86 requests per second).

So this is not a replacement for the most common use cases for Elasticache. It is interesting as a cache in front of S3, but if you want a cache in front of S3...just use Cloudfront?

Honestly if I was AWS I would be ecstatic if people used this, AWS probably breaks even or maybe loses a little bit on optimal usage of this, and makes an absolute killing if someone using this has a traffic spike and ends up hitting their lambda millions of times.

So it uses serverless functions, but still requires EC2. “ deploy InfiniCache on a EC2 instance with powerful CPU and high bandwidth”

I don’t understand.

EC2 runs proxy, which does erasure coding and lambda periodic pings to keep them running and therefore preserving cache.

So basically abusing how serverless functions are supposed to work by "warming them up". I wouldn't count on this working long-term as cloud vendors will be more aggressive in discouraging this behaviour.

Eventually it gives them an incentive to embed such functionalities out of the box if they notice significant usage pattern. Why fight it when you can embrace it and profit from it?

Long before AWS implemented provisioned capacity, these kinds of hacks were pretty common.

I would expect that the number of customers that would roll this out and run it without any mistakes that drive up the monthly spend is small enough that Amazon doesn’t care.

I like the hacking aspect of their solution. It's always interesting to see products and services used creatively.

It's always interesting to see terminology evolve. "Serverless" started as a hosting terminology, to mean "somebody else's servers". Here, it's still being used to mean "somebody else's servers", but in a domain that could reasonably use "serverless" to mean "without a server" instead. Caching to a local hard drive is, almost by definition, serverless.

It's a very neat application, but the terminology erks me.

"Serverless" was a company name, IIRC. Their model caught on and was hijacked by the community. Now it's one of those nebulous buzzwords that means everything and nothing.

Yes, Serverless Inc. creators of the serverless framework (but it went the other direction). The framework is an open source tool that you can use to "deploy" "serverless" assets. Basically it covers most of those scripts you usually have to write and gives you a declarative format with some great variable capabilities that help you managed config and declare your components clearly. Running the tool deploys to your cloud of choice and results in functions, APIs, workflows, databases, or whatever you declare now existing and ready for use.

Far more seriously in usage, the framework is separate from the class of services regarded as serverless. The star that kicked it off was AWS Lambda. Serverless colloquially means you are dealing with an abstract service contract rather than a server (e.g. the oldest is S3). This removes patching and other maintenance that usually does not directly support business value. More formally, serverless includes auto-scaling to zero, paying only for what you use, high availability, and other design patterns most outside the large tech houses cannot use at low to no cost.

Full disclosure: I contributed to the serverless framework. I'm a biased fanboy.

Presentation (slides+video):

https://www.usenix.org/conference/fast20/presentation/wang-a...

InfiniCache: Exploiting Ephemeral Serverless Functions to Build a Cost-Effective Memory Cache

The paper name fits the title better.

The keyword here is 'Cost-Effective'.

Very creative, extremely complex, what a fun hack. TL;DR exploit the fact that AWS keeps your lambda process running (and in memory) beyond the time you are billed for in order to build an ephemeral cache. Creates one lambda per slice of cache (think redis slots) and provides a proxy to hide the (substantial) complexity from clients, keeping those lambdas warm, etc. I could see this being useful for cost hacking if the request volume was on the order of one request per key every few minutes.

Reply on news.ycombinator.com