key = calculate_cache_key()
if not cache.has(key):
data = expensive_calculation()
cache.store(key, data)
else:
data = cache.get(key)By default it handles the case of concurrent retrieval on the same key (the second one will just wait for the first one to finish and use that value rather than starting a duplicate computation). It also lets you configure more interesting things like eviction strategies, removal notifications, and statistics.
Last year was a lesson for me in why caches are a hard problem as I had to debug many cache issues from other people not thinking things through... (At least one of the issues was my own fault. :)) Since then whenever someone suggests we use a cache I instinctively pull out a set of questions[0] to ask. The three questions Guava has you consider can also lead you to using memcached or the like instead but my set tries to answer the question "Do you even need a cache?" and if so, generating helpful design documentation.
Is the code path as fast as it can possibly be without the cache? Do you have numbers?
Will the cache have a maximum size, or could it grow without bound?
If it grows without bound, either because of unbounded entries or because of unbounded memory for any particular entry, under what conditions will it consume all system memory?
If it has a maximum size, how does it evict things when it reaches that size?
Are you trying to cache something that could change?
If so, how is the cache invalidated?
How can it be invalidated / evicted manually by another thread or signal? (Debuggability, testability, inspectability/monitorability, hit rate and other statistics?)
Is there a race condition possibility for concurrent stores, retrieves, and evicts?
How constrained are your cache keys, that is, what does it need to know about to create one?
Do they need to take into account global info?
Do they need to take into account contextual information (like organization ID if your application server runs in a multi-tenant system, or user ID, or browser type, or requested-language)?
Or do they only depend on direct inputs to that code path?
To frame it another way, what happens if 100 requests come in at the same time for that expensive value when it isn’t in the cache yet? The expensive calculation will be run 100 times at the same time.
Ideally, you’d rather refresh the cache value in the background once and never allow duplicate requests for it from the web.
If you’re running a language that makes it easier to deduplicate requests for certain data, the original approach will last longer. The CacheEx library in Elixir, for example, will only run the expensive calculation once, set the cache and then send the value back to everything that requested it while it was loading.
CacheEx gets the ability to check for the presence of the cache key in Erlang Term Storage (ETS) which is basically an in-memory cache. If the key is present, it just returns the value.
If it's not, it sends checks to see if a process exists with the cache key name. If there isn't one, it creates one to request the resource.
For any other requests that come in until the value has been created, they will be directed to the process that is getting the value.
When the process that was calculating things comes back, it will save the value to ETS and then also send it back to all of the queued processes that have been waiting for it.
In the case of Varnish, you'd be expecting it to send back the entire completed view...HTML and all. This isn't something that you need to worry about with Elixir because the view is never actually rendered in the application. It's broken down into pieces that are never duplicated in memory and then replayed directly to the socket...meaning you really only ever need to cache expensive data and not what it's transformed into.
Here's a good read on why this view layer is so fast, if you're curious. Most people report shock that their uncached performance with Elixir and Phoenix is on par with statically cached HTML. I didn't believe it until I saw it myself.
https://www.bignerdranch.com/blog/elixir-and-io-lists-part-2...
For larger, more complex scenarios the techniques mentioned by GGP above work well, i.e. not doing expensive_calculation() in your app process at all.
The issue has a bunch of names, I know at least three: cache stampede, thundering herd and dogpile effect.
This gets more complicated if you have a distributed cache and/or distributed application servers. The typical solution there is to allow at most one computation per process/device.