Hacker News Reader

Comment by hinkley | Hacker News Reader

Caching is a weapon of last resort. In most cases it is the last major performance improvement you will see from your code, even if other options were available, and any information architecture problems you have are set in stone or will get worse.

Because of the locality aspect, people will assume that it's 'free' to look up a value multiple times instead of passing these data dependencies between the relevant methods. This turns your cache into global shared state, and once that happens you can't turn the cache off again. There are other problems with this arrangement as well.

The easiest to understand (and yet I've failed on at least one project) is that if the working set for a problem ever becomes proportional to the size of the cache (>1 for single task systems, > 1/n for systems with n concurrent tasks), then you can end up evicting data in the middle of a transaction. The obvious consequence is that it causes performance to jump off a cliff. Particularly bad behavior with LRUs but switching to a different statistical model simply reduces the height of the cliff. There is still a cliff. The less obvious consequence is now you have concurrency bugs, because the code assumes that decisions made about the data will go the same way on each call, but due to eviction, two halves of the transaction may make conflicting decisions with undefined behavior.

But speaking as the guy who fixes the perf issues other people gave up on: Once you introduce caching you poison the performance well. Flame charts lie when cache is used, and after people rely on the cache it is lying in the other direction when the cache is off, because you're calling that path 5 times now and it's swamping other bottlenecks. This is what I really mean when I say it's a weapon of last resort. Because once you use it, it attempts to take any other options off the table. Caching will be most of the last order of magnitude in scalability that your system ever sees. Every 'win' from there on will take more time and energy than most people are willing to invest.

View on HN

I'm not sure everyone here is talking about the same thing when they talk about caching but if you took caching out of the modern Internet it would collapse immediately. There's caches in your CPU, there's caches in your disk drives, there's caches in Linux, there's caches in every layer of every major cloud service, there's caches in your network (CDN).

If your CPU had to fetch every instruction and every bit of data from main memory it would run x100 slower. If your disks didn't have caching they'd generally run slower. If Linux didn't cache your filesystem it'd need a lot more disk accesses. etc. etc.

Ofcourse a properly designed cache has to consider things like what are the access patterns, how to figure out what to keep in the cache and what to toss away, how large is the dataset we're caching, invalidation etc. But there's no performance optimization that's going to get that NetFlix movie to 100M subscribers directly from the disk it's stored on, through the network.

Feels like some of the discussion is about something like storing f(x) in a map instead of computing f(x). I wouldn't use this terminology (caching) in this case. This is more like your standard space to time algorithmic tradeoffs.

And then there’s Spectre and Meltdown.

Here we are talking about building your own cache data store, not the ones built into your processor.

My first impulse was to disagree with you, but I realised I gave the exact same advice as a consultant to a group managing one of the bigger government websites that had some performance issues.

They wanted to paper over the issues by slapping a CDN in front of the site. Their users are all local, so the global aspect of the CDN wasn't the point, the benefit was that it would cache the content and provide a performance boost.

I told them that, yes, a CDN might help, but they should optimise their site first, and only then add a CDN on top if they really need to. Otherwise they'll just mask their issues. It's too easy to end up with queries that take 1 minute but are cached for, say, 1000 separate users. Most people most of the time will have a good experience, but you'll end up with a hideous 99th percentile that will be really hard to detect and fix.

I’d often take the opposite approach (and have done with consulting clients)

Slap the CDN on the site and then chase down the slow server responses

It delivers an improved experience for the people using the site sooner than optimizing the site first

The challenge of course is you then have to persuade people to go chase down the slow responses when they may response “there’s so few why should I care?”

From a web standpoint I reminded them that the browser already has a cache so you would be better off getting your cache semantics clear first, debug it at the client and then infill later if and as necessary.

In my extensive experience of optimising stuff...

> Caching is a weapon of last resort. In most cases it is the last major performance improvement

...is wrong. It is one of the very first things I reach for because it is the most effective and simplest, once you've eliminated O(n^2) behaviour etc. It's not a magic wand but nothing is.

That's not a very convincing argument. "I use it because it works" without refuting any reasons I presented as to why it "works" but breaks your other options.

How do you teach people to effectively use a profiler on code with pernicious caching?

Part of why I'm the "Cleaner" is because I've worked with any number of people who will present "evidence" that everything that can be done has been done. Here are the metrics from a year ago when we turned on the cache. Here are the flame charts. See? Nothing to see here. They get really uncomfortable six months later when I'm reporting 2-3x in code that can't be optimized further. And in each case everything was made more difficult by the people ahead of me chumming the waters.

They were partially correct. There was in fact nothing left for them to see, so nothing that they knew to do. But "there's nothing to see here" and "there's nothing here" are two very, very different statements.

The issue with caches is they are easy to apply. You can easily drop a cache in any place you load a resource without significant code changes.

I tend to think they are ok weapons of sometimes first resort (rarely changing resource with heavy system impact). But you'll definitely get better results simply by rethinking and rearchitecting stuff. (Did you really need to load the database just to lookup a single value?)

> You can easily drop a cache in any place you load a resource without significant code changes.

Only when the cache is brand new. The problem with people with cache addiction is that they think they're free. So once they know a cache is there, they begin to lean on it, and lean on it. I've seen it with many different teams in a number of verticals. Sometimes while I'm there, sometimes before I arrived. It's the people not the company.

This illusion that you can always turn it off later is dangerous. That's only true if you have an impeccable information architecture when you start. And if you have that, you've already done a lot of heavy lifting before caching becomes your only option.

It's dificult to refute your argument because I don't understand your points:

> Because of the locality aspect, people will assume that it's 'free' to look up a value multiple times instead of passing these data dependencies between the relevant methods

what does that even mean? What does 'data dependencies' mean specicially?

> then you can end up evicting data in the middle of a transaction

So in the middle of a transaction, instead of looking up data from a key - very cheap - you should do ... what? Recalculate it instead?

> The less obvious consequence is now you have concurrency bugs, because the code assumes that decisions made about the data will go the same way on each call, but due to eviction, two halves of the transaction may make conflicting decisions with undefined behavior

Assuming I even understand this, you are saying where there's concurrencty there's a risk of pulling stale data out of a cache. Well, true. That's just something you have to be careful of. Add a Tx reference to the cache key.

> Once you introduce caching you poison the performance well.

Not my experience at all. I can't understand why you're saying this.

> Flame charts lie when cache is used

How so?

> and after people rely on the cache it is lying in the other direction when the cache is off, because you're calling that path 5 times now and it's swamping other bottlenecks

I've no idea what this means

> This is what I really mean when I say it's a weapon of last resort. Because once you use it, it attempts to take any other options off the table

Heh! Absolutely not! Other optimisations are orthogonal to caching IME:

Better algo - orthogonal

Go lower level - orthogonal

Go multicore - orthogonal

Noobish SQL query is begging for a rewrite - orthogonal

Use L1/2/3/ chip cache better, or RAM, by careful data layout - orthogonal

Use a better library - orthogonal

Use better data structure eg. than linked list - orthogonal

>> Because of the locality aspect, people will assume that it's 'free' to look up a value multiple times instead of passing these data dependencies between the relevant methods

> what does that even mean? What does 'data dependencies' mean specicially?

It means that instead of having a method like `entryPoint(int)` that looks up a `Foo` by id and then passes the `Foo` around (`verify(Foo)`, `Foo update(Foo)`) you instead wind up with a system that passes around an `int` (`verify(int)`, `/* Unused */ Foo update(int)`) and each layer then looks up `Foo` in the `FooService` (because `FooService` has a cache and subsequent lookups should be essentially-free).

Even worse when `FooService` is abstracted behind a thread-local like `ThreadLocal<Foo> currentFoo`, the implementation of which is `fooService.findById(currentFooId.get()) /* cheap, because fooService has a cache */`.

My own take on this is that caches are global-by-default when you really want local-by-default (e. g. dynamic programming instead of a `static final Cache theCacheForTheEntireProgram`). Local-by-default caches are easy to make transactional (when the call stack returns the cache is gone) and don't grow in an unbounded manner in the general case.

Yes!

Local by default also has the feature/problem of making you look at the scale of the data involved in your query up front, before it's having a 100-1000 req/s hitting it in production. It's harder to pretend this large dependency graph will just magically take care of itself (or worse, be someone else's problem after you've declared victory) when it's staring you in the face.

There's a somewhat tangentially related heuristic to your points imho. When writing code, when in doubt, task CPU over memory, it will generally yield leaner, better software. Easier to profile, debug, and optimize. Much easier to scale in production environments too.

Code which optimizes for fast running time, tends to lean heavily into using memory-bloating data structures and caches. These are often poison for aggregate performance in real world environments, and like you said among other things, reduce visibility in the workings of system.

There's some implications of this stance; e.g. optimizations to reduce memory footprint are generally never wasted, while optimizations to squeeze out a few ns of runtime may have many more unexpected consequences.

> How do you teach people to effectively use a profiler on code with pernicious caching?

Profile the function that is called if object is not found in cache ? Have pluggable cache with plugin that just doesn't cache? That is not hard problem to solve.

> They get really uncomfortable six months later when I'm reporting 2-3x in code that can't be optimized further. And in each case everything was made more difficult by the people ahead of me chumming the waters

Well, they delivered (I'm guessing) 10x improvement in a month by adding cache. Caching is entirely reasonable first step and it can be good enough to be last one.

And frankly ability to use profiler and knowing some architectural optimizations seems to be pretty rare skill.

Exactly! People here seem to be making a complex thing out of a simple thing.

What makes caching one of the hardest parts of computing is this assertion that it's easy. Everything is easy when you oversimplify it. Some things explode in complexity when you try to force them to be simple.

  > Some things explode in complexity when you try to force them to be simple.

do you mean to say iow "simple in the small but when assembled (in the large) is more complex" whereas what we should be aiming for is perhaps "a little more complex in the small but with the benefit that it enables a simpler (and probably more scalable) architecture once assembled (in the large)"??

> Profile the function that is called if object is not found in cache ? Have pluggable cache with plugin that just doesn't cache? That is not hard problem to solve.

The more the dynamic behavior of the profiled system and the real system deviate, the harder it is to isolate what's causing the slowdowns.

> Well, they delivered (I'm guessing) 10x improvement in a month by adding cache.

Yes, but by doing this they also tremendously increased the complexity of the project, and (unless the team has someone like GP in it) prevented further optimizations. It may still be a good trade-off, but the overall sorry state of pretty much all software everywhere suggests it may not be a good one usually.

> And frankly ability to use profiler and knowing some architectural optimizations seems to be pretty rare skill.

That's kind of reinforcing the parent's point, though.

> The more the dynamic behavior of the profiled system and the real system deviate

Maybe. I've never found it a problem, but maybe. When I've added a cache, the cache comes with instrumentation for exactly that.

> Yes, but by doing this they also tremendously increased the complexity of the project

It depends. EG if you don't need to worry too much about cache invalidation, adding caching can be as simple as adding a single function call eg like python's lru_cache. It depends on the use case.

>> Profile the function that is called if object is not found in cache ? Have pluggable cache with plugin that just doesn't cache? That is not hard problem to solve.

>The more the dynamic behavior of the profiled system and the real system deviate, the harder it is to isolate what's causing the slowdowns.

Whose quote you repeat without thinking ? It's not that relevant if you know how profiling works...

>> Well, they delivered (I'm guessing) 10x improvement in a month by adding cache.

>Yes, but by doing this they also tremendously increased the complexity of the project

Not really in most cases its pretty simple to add it, and if you profiled correctly you'd start with only adding extra code in hottest part of the app

> and (unless the team has someone like GP in it) prevented further optimizations.

Uh, just no. It might've fixed problem well enough to not be touched but it ain't stopping anything, just a different solution.

> It may still be a good trade-off, but the overall sorry state of pretty much all software everywhere suggests it may not be a good one usually.

The assumption here that the same developers that fucked up caching would somehow manage to achieve same speedup with "just" optimizations and less bugs is bold one.

Hell, many low level optimizations are also basically "using cache" (but CPU one) well, althought compiler usually gets you out of most egregious mistakes (unless you decide to write multithreaded C/C++ carelessly I guess.

>> And frankly ability to use profiler and knowing some architectural optimizations seems to be pretty rare skill.

> That's kind of reinforcing the parent's point, though.

What point exactly ? That average developer is bad at their job ? Poster above explicitly said that the worse developers claimed it can't be optimized more in the first place, what else you expect from them ? Cache was the most cost-efficient solution most likely.

> Profile the function that is called if object is not found in cache ? Have pluggable cache with plugin that just doesn't cache? That is not hard problem to solve

Okay, now I have a profile that says 90% of the time is spent in the code the cache avoids. Now what? Narrator's note: When the cache was put in place it accounted for 40% of run time, but now people use it because it's 'free' and so the code has grown from looking up the data twice to looking it up five to seven times depending on conditional logic.

> Well, they delivered (I'm guessing) 10x improvement in a month by adding cache.

No, they didn't. That's the problem. No cache gets you 10x. 10x for one step of a 20 step process is a positive improvement but rarely more than 2x for the first one, 4x at the outside. If it does more that that then someone is being titanically stupid. From then on each cache is going to net you less, and less, and less. Only if someone measured wrong or you are blocked by some architecture change that's needed so that caches even make sense, will you see a bigger improvement this year than you did last.

But the pathology I'm discussing above didn't even have anything at all to do with caching. It's the psychology of (ir)rationalizing around performance, the assertion that you've done everything that can be done because you've done everything you know to do. It's a bizarre love child of learned helplessness and ego self defense. It's gotten way worse in a world where people reach for caches first (and thus, I assert, last), but it was always there.

I have a couple coworkers who are trying to be clever with a new cache architecture. Irritatingly by taking a statement of fact made by me 3+ years ago and jumping to the wrong conclusion. They promised 50% in 18 months (I was suggesting we target 90% in three years). Now we are somewhere past 30 months and so far they've delivered lower numbers than I did in 5 man weeks spread out from Christmas to mid March, while juggling other priorities and suffering from Seasonal Affective Disorder. Another coworker, usually cache averse, was making noises about adding a cache for some other pain point, and I started pulling on the thread for a bit until he was satisfied. It'll be next March before they surpass the 20% I turned in with about 8 modest changes. And the architectural improvement from both efforts will be roughly on a par. Top down is sexy, bottom up is not. But it's hard to do the former without the latter. Eat your vegetables.

> No cache gets you 10x

I've done much better. Think sticking a a cache in front of a database with seriously complex queries. Yes you need to be be aware of staleness, no it's not a magic wand, but it delivered massive improvements.

> When the cache was put in place it accounted for 40% of run time

If looking shit up in a hash table, at a few hundred or thousand cpu ticks, cost 40% of your runtime then I smell PEBKAC[1], not cache problems.

[1] Not you personally but someone

>> Well, they delivered (I'm guessing) 10x improvement in a month by adding cache

>No, they didn't. That's the problem. No cache gets you 10x.

I was actually being conservative. We had almost 100x increase in some cases.

It was pretty particular app, site took few hundred (IIRC closer to second mark) ms to generate before we started at all.

First it was simple Varnish cache but that had problems, some components never changed, other were updated pretty often (stuff like "current users here")

First it was split to various components, each with different TTLs, then all of that was merged in Varnish ESI.

So you could say invalidate a part of the site and varnish would

* load this component from app server

* load the template it contains from its own cache

* render it.

* no refresh would be just direct from cache, single digit ms or lower latency

* refreshing one part of site would have latency of rendering only that tiny part of site on app server

later we also updated that with "grace" feature of Varnish where you could define time interval where cache would be valid but generate request to the backend to refresh

So if said low TTL component was refreshed often

* the component will always go from cache with lowest latency

* if it was on its last seconds varnish would send the request in the background and refresh cache

Now the app itself wasn't very well written in the first place (there is a book of horror stories to write about it) but they (we did ops and testing, they did the app stuff) spent less time than "fixing it properly" and instead rewrote it to a bit more modern architecture. They still use same cache to cache JSON responses to API endpoints tho.

That not even to say how much of DB performance is actually clever caching, after all index is just pre-generated partial cache...

But sure, some apps are better for it than other, 10x is definitely not universal.

> From then on each cache is going to net you less, and less, and less. Only if someone measured wrong or you are blocked by some architecture change that's needed so that caches even make sense, will you see a bigger improvement this year than you did last.

Well, doh, caching something you cached already ain't gonna be that effective. But if you see same request with same done 100 times in a minute, making it take 20ms instead of 100m ain't gonna be as effective as making it take 100us to get from cache. You might want to address it eventually to have lower jitter, but same cache can allow you to make 20 other methods faster on average and that is less code and effort than optimizing 20 other functions..

> I have a couple coworkers who are trying to be clever with a new cache architecture. Irritatingly by taking a statement of fact made by me 3+ years ago and jumping to the wrong conclusion. They promised 50% in 18 months (I was suggesting we target 90% in three years). Now we are somewhere past 30 months and so far they've delivered lower numbers than I did in 5 man weeks spread out from Christmas to mid March, while juggling other priorities and suffering from Seasonal Affective Disorder.

And do you think they would be able to instead go and optimize it better in that time ? That just sounds like incompetent co-worker and not anything to do with cache...

That guy just keeps repeating "it's hard" then 'explaining' why in obtuse terms and in ways that don't stand up (I don't think he understands database behaviour). Frankly I don't think he has much experience in this area. I'm going to give up now, he's spent a fair bit of your and my time and still can't accept he's not right.

I kinda feel that your convincing counterargument is just ‘I have a different experience’.

I’ve never seen people reach for caches first, and then discard all further optimizations.

Or rather, I’ve seen them do that, but not out of some misplaced feeling that it was the only solution, just the one that was easiest to accomplish in very little time.

Interested in any resources that you've found vital in your journey.

I'm angling to go deeper into the "Cleaner" path, and it's a very slow "figure it out as you go" experience.

Caching can turn O(n²) algorithms into much faster algorithms. I would point to how transformative tabling is in Prolog, not to mention memoization rescuing the calculation of fibonacci numbers by recursion, which otherwise would be gross malpractice.

These are very well-behaved instances of caching. They are so well-behaved that they can be fully described with a little bit of analysis. Once I/O is involved or the load isn't well-known beforehand, it becomes much more difficult. Lazy I/O in Haskell is a very divisive topic for a reason.

Those aren't caching. They're dynamic programming. There's a profound difference and it's about scope and reachability.

Edit: and about who's driving.

Memoization is caching and a form of dynamic programming; thus caching is sometimes an important tool in dynamic programming.

But the fact that caching can be useful, even essential if your scenario demands such a clear-cut space-time tradeoff as in memoization, seems beside your point.

Caching is fine is there's no state involved. e.g: pure functions like the fibonacci sequence.

As soon as your cache depends on a mutable state, things get really hairy. Cache invalidation is one of the 2 hard problems in computer science, after all.

If it's really no state it should be a compilation step.

After watching some coworkers play keystone cops for 5 years I'm preparing to do just that to the last project they were responsible for. Their new project should have been that as well, but no it's going to be cache.

This gives me flashbacks to Intel's great disaster of Itanium which seems so ridiculous in retrospect but I guess very few people understood superscalar processors in the 1990s.

I did some serious thinking about processors that use the transport triggered architecture

https://en.wikipedia.org/wiki/Transport_triggered_architectu...

which are a simple way to make a custom CPU (for the first time I thought, "I can make a custom CPU!") but do operations in lockstep in such a way that the whole thing blocks if a memory operation stalls. You might be able to get away with however because the TTA is meant to be codesigned with the code it runs on.

Itanium was smarter than a TTA but it was predicated on the idea that a compiler could know how operations can be interleaved at run time. The compiler has no visibility into what gets cached, but at runtime the processor can opportunistically run operations that aren't stalled. For a general-purpose processor that's supposed to run general code, conventional superscalar processors do better than VLIW processors that require compilers to guess ahead of time what is impossible to guess.

VLIW works _really_ well on a deterministic ISA. As in when you know how long a load is going to take at compile time because you know it's N clock cycles from that type of memory.

Otherwise I really like the GPU derivative form where N instructions are ready to go and the hardware picks one where the prerequisite memory access has completed, despite that being variable.

> VLIW works _really_ well on a deterministic ISA

that's a very good point.

> Otherwise I really like the GPU derivative form where N instructions are ready to go and the hardware picks one where the prerequisite memory access has completed, despite that being variable

Hmm, isn't that OOO (out of order) execution? And isn't that something GPUs explicitly don't do? (AFAIUI)

I'm probably describing it poorly. A GPU (at least the AMD ones) has N execution units (of a few different types) and M threads (wavefronts) waiting to be scheduled onto them. Each cycle (ish), hardware picks one of the M threads, runs it for an instruction, suspends it again. Memory stalls / waits are handled by not scheduling the corresponding thread until it's ready. The operation might be a CAS over pcie so it's really variable in latency.

Provided you have enough (ideally independent) work item / task / coroutine things available to enqueue, and you're designing for throughput instead of latency, that keeps the execution units or memory bandwidth fully occupied. Thus win.

The x64 style model is deep pipeline + branch predict / speculation and it all goes slow when the guess fails. In contrast the GPU doesn't have to worry about guessing a branch destination incorrectly and unwinding wasted work. It does have to worry about having enough work queued up, and distribution of tasks across the compute unit / symmetric multiprocessor units, and the hardware-picks-the-order leaks into the concurrency model a bit.

Nice clear answer, thanks.

Wouldn’t this whole problem be solved by a graph-based architecture, used already by some compilers as an intermediate? Then the CPU sees where and when everything is needed and schedule the loading of registers as optimally as it is feasible, at runtime.

Well, usually memoization gets you polynomial time from exponential time with the space tradeoff. It's unusual to start with `O(n^2)` and get a better DP solution.

In my extensive experience of optimizing stuff it is right.

I cannot even count how many issues were created downstream of the decision to start caching X, Y, or Z. At a point, every other software engineer gets the bright idea to cache their special little function or request or database call. Soon enough, there are N layers of cache responsible for delivering some special piece of data. The engineers build tooling to clear those caches on updates, either manually or automatically. Both suffer from issues, the latter occasionally failing to perform and just causing general confusion about why stale data is returning, and the former being far worse, recruiting dozens of business stakeholders into the problem of knowing-how-to-expire-cache. They start making their own internal wiki pages with steps or various cache tools to clear, often incorrect steps because they got lucky one day by clearing cache X and not cache Y that they did not know about.

Overall, in my experience, allowing software engineers to do caching as they wish without any real guardrails has caused massive problems most likely on the order of N!. I cannot roll my eyes far enough into the back of my head when I hear someone suggest caching as a solution any longer, at least not on an application that has more than a dozen business stakeholders managing data and expecting it to be updateable in any reasonable way so that customers see the most recent data when the business wants it.

I have one particularly baroque data path that I think passes through three layers of cache. The same people worked on it the whole time, and I'm 90% sure that at least one of those layers is doing absolutely nothing, except making the code harder to read. Once they're added nobody takes them out. Like all global state it's problematic to be absolutely sure it's not being used anywhere.

If I had to pick the two worst sources of bugs I can imagine, it would be (1) concurrency and (2) caches. Those functions often require extensive performance and load testing and have tons of non-deterministic failure modes.

No way, the worst two sources of bugs I can imagine are definitely (1) conc(2) cachesurrency and (3) caches

This hurt to read. Like, in my heart.

Try CNN subtitles on YouTube. They are almost always data race gibberish.

The classic joke is to make (3) off-by-one errors

Indeed I was trying to make the duplicate entry an off by one error without explicitly saying so, but I suppose I should have for clarity

I like applying the Circuit Breaker approach to caching. Rather than applying a cache everywhere, setup a cache wrapper that only triggers if the accessed resource is slower than a certain threshold.

The goal is for the cache to go unused most of the time, but during times of unexpected system stress the caches will kick in, provide relief to the system and reduce impact to users.

Having a different code path that only triggers during high load feels pretty risky to me.

In theory, the best way to manage this situation may be with a grayzone - if all is well, then there's only a (say) 5% chance of going to cache. (To at least catch "the cache is broken" errors sooner.) As things get worse, that chance goes up as you cross-fade to mostly using the cache.

Important Warning: If you do not have a very good understanding of your overall system, or neglect to carefully dampen the cross-fade functions - then the whole thing can demonstrate fast and unpredictable swings in behavior. Note that being subpoenaed to testify in a "Why did the Crater City chemical plant blow up?" investigation is usually bad for one's career.

That's better but composition is always the devil in the details. Which might be what you mean when you say:

> or neglect to carefully dampen the cross-fade functions

People with a 'caching problem' don't have one cache, they have a bunch of them. So that 1 in 20 chance of getting grey zoned on one cache, is 1 in 400 chance of getting grey zoned on two caches, and a 1 in 8000 chance to get grey zoned on 3. That is a very small fraction but when you're dealing with the Law of Large Numbers that's something that's happening at least once a second for many services.

But if you're using this you probably have a lot more than 3 caches, and then we have to talk about the odds that any request gets grey zoned at least once. which is going to be 1 - (19/20)^n. At six caches you're already over 1 in 4, which is going to potentially change your mean response time quite a bit.

> "...which is going to potentially change your mean response time quite a bit."

My experience is that the changing mean response time isn't nearly as bad as the changing standard deviation.

The really big problems tend to arise when you have unsuspected or under-appreciated positive feedback loops - whether in the software, or in associated physical systems and human behaviors. Or resonance effects. Those situations are where good dampening gets really important.

> The goal is for the cache to go unused most of the time

Then you have a useful cache sitting there unused unless at peak time - why? Just enable it always. This makes no sense.

Depends on how large the cache is and how much users expect instant updates to their data. If real-time or near real-time is the expectation, then avoiding the cache usage unless necessary preserves their experience without incurring the hurdles of cache invalidation that grow more complex the more you cache.

EDIT:

Another area where this approach is beneficial is with a multi-tenant system where larger tenants querying more data could have a much more significant performance impact than other tenants on the system.

Time of day/week/month usage spikes can also cause a performance degrade for everyone even if it's not typically stressing the system.

This approach is just a mitigation to prevent the system from getting overload during those times. It's not the right approach for every situation. Just a spin on the Circuit Breaker Pattern.

https://en.wikipedia.org/wiki/Circuit_breaker_design_pattern

I'm beginning to wonder whether my personal moniker of "signal propagation" is nominatively equivalent to what others call "cache invalidation".

I.e. you cannot by definition have a state change until the signal thereof has traversed a link from A to B. No signal, no change. If you have a cache for one system that delays writes to a backing data store referenced by other systems that are unaware of that cache, then you have necessarily a partition problem.

Every problem in a distributed computing context can generally be reduced to a signal propagation problem.

It's only a problem if you're breaking deadlines. Like having someone's status in communication app update with 10s delay is probably not a a problem, so as long as you don't ever cache it for longer than that, it is entirely fine.

> If you have a cache for one system that delays writes to a backing data store referenced by other systems that are unaware of that cache, then you have necessarily a partition problem.

In most cases people mean caching reads so the problem here is stale reads, not delayed writes. It's not necessarily the same, some component caching (or essentially, queuing) write means that once the write is done, all readers (assuming they don't have any extra cache) see it at once, while with caching reads (and no invalidation), multiple readers can have different view

The one that really jams me (and others) up is that signal propagation and transaction lifetimes line up badly. Caches, being cheap imitations of ACID 'compliant' databases, which themselves cheat all day every day to make pretty numbers, almost never have read repeatable semantics, so if your code splits responsibility for fetching a piece of data you have one of the problems I complained about at/near the top level: Half of a decision can be made with one version of a piece of data and the other half with another, resulting in undefined behavior. In a system that asserts that A = !B you can have an interaction that did some of the activities for A and some of the ones for B (because we read A and A' and assumed they were identical).

You're making a lot of assertions without any backup, and they don't make much sense.

> Caches, being cheap imitations of ACID 'compliant' databases

No they're not, not remotely.

> which themselves cheat all day every day

I don't believe a word of this, have you any actual evidence (other than I believe Oracle's serialisable is actually snapshot isolation IIRC)

> Half of a decision can be made with one version of a piece of data and the other half with another, resulting in undefined behavior. In a system that asserts that A = !B you can have an interaction that did some of the activities for A and some of the ones for B

And this just makes no sense, you're claiming that databases are not producing the results they should, if that's the case then I don't believe you understand transaction isolation levels or indeed how database queries work, in which case you have problems other than and larger than caching.

This whole thread s just getting weirder.

As someone said in another subthread:

> That's with everything that have any level of complexity, just see how many people got transaction isolation wrong just because they had some simplistic view on how it works in database they use.

Any database without MVCC works quite a bit differently than people think it does. And ones with MVCC run into the ceiling if you assume they work like databases that aren't based on MVCC. Like leaving long transactions running in the background.

The problem is, like caching, difficult to catch in a petri dish.

> This whole thread s just getting weirder.

Maybe you should take a break.

If you assume your database isn't doing much for you, you

You can queue cache refresh in the background if the object in cache is close to the "limit" of how fresh you want it to be. Need a bit of cleverness to avoid thundering herd problem but otherwise works pretty well.

Also if you really want near-real time you might want to actually push the updates directly to cache instead of relying client to trigger cache filling. cases where you need a ton of data and it served real-time and it's same object that constantly changes (and not say pointer to a position of video stream) are pretty rare

My problem with the term 'cache' here is that people have expectations about what cache is and when you violate them you tend to discover that they've returned the favor.

> push the updates directly to cache instead of relying client to trigger cache filling

You can use cache software to fulfill such a role but it's not exactly a cache anymore, and keeping people from treating it like one is some kind of cat herding.

You're using it as a data store at this point, which for one has very different eviction semantics - exactly 1 copy of each concept, no more, no less. You can store such things in Consul, for instance, and get very low stale read latency. The cache implementation scales higher but breaks sooner. Pick your poison.

Well, it depends on your data model. Like, if you write only a little (compared to reads), write-thru model of cache (write to cache as well as to backing store) will lose you tiny % of hit ratio at profit of never having to worry about stale data.

> My problem with the term 'cache' here is that people have expectations about what cache is and when you violate them you tend to discover that they've returned the favor.

That's with everything that have any level of complexity, just see how many people got transaction isolation wrong just because they had some simplistic view on how it works in database they use.

I've always been a bit of a fan of debounce for this but many of the obvious implementations are cache based which is essentially inviting people to do the thing I just said I hate.

The cost of counting upvotes or logged in users is not worth the incremental value it affords the user. These are examples of situations where 'eventual consistency' can be embarrassingly easy. If you split the source of truth (SoT) from the system of record (SoR), then the cost of updating the 'truth' can be amortized across the writes to the SoR. The reads do a straight lookup from the SoT, which was the truth at some point but is now the fiction everyone is agreeing to. The distinction is that everyone agrees to the same source of truth. Nobody ever goes asking the SoR for a second opinion, except the SoR, which updates the SoT when appropriate. The SoT is dumb, the clients are dumber, only the code that manages the SoR has any smarts to it.

Read traffic can escalate pretty high in such a system without much stress.

Exactly. Another situation is a multitenant system where some tenants are much larger than others, triggering larger queries which potentially have a larger performance impact.

WTF?

> If real-time or near real-time is the expectation, then avoiding the cache usage

Because looking up a precalculated result in a hash table is so much slower than not looking it up? And what, recalculating it instead is faster than an O(1) hash table lookup? If so,why are you caching?

I am just getting more confused by what people are saying here.

> I am just getting more confused by what people are saying here.

Maybe the takeaway is that this shit is hard and everyone is a little bit confused in their own way.

I'm not saying the only way to win is not to play, but I can neither confirm nor deny rumors that I may have at some point muttered this under my breath.

I do think you've hit on a point of confusion in the parent, but...

In a (hard) real-time system the only thing that matters is the worst-case execution time -- if your worst-case time fits in the allowance you win, if it doesn't you lose.

Adding a cache improves the average case (typically), but hurts the worst case (always) -- your worst case goes from "do expensive operation" to "do lookup in cache, fail, do expensive operation." Even if you parallelize misses ("do lookup in cache | do expensive operation, on cache success cancel expensive operation, on cache miss wait for expensive operation" you're generally adding at least a few operations in the critical path.

You raise a good point but I would never have thought caching appropriate to an RTS, not that I have any experience with those. But yes.

Because it's meant as a safety net, not standard operating procedure.

Related idea: if your system is running close to capacity, or at capacity, you're one hiccup of activity from disaster.

> if your system is running close to capacity, or at capacity, you're one hiccup of activity from disaster.

oh man if i could scream this to the roof tops i would

most people (esp managers!) just do not get this...

Sounds more like a dark buffer anti pattern

...instead of having better user experience all the time ? And have more code to manage on top of that ?

Sorry but that sounds like terrible idea.

And when system starts being slow the cache is empty, because you waited for system to slow down to start filling it

If the system is fast enough without the cache this isn’t a problem. In order to overwhelm it, you need a cascade of stress at the same time.

Caching as the stress appears prevents the issue.

Databases already do plenty of in RAM caching with available free memory anyway, you just don’t get to manage it directly.

Do cpu caches poison the performance well?

Everything nondeterministic adds difficulty to the Scientific Method. Maybe we'll have better ways to filter the noise in the future, I certainly hope so, but that's not where we are now. But this is a little whataboutism.

Caches in the CPU or anywhere else are a leaky abstraction, and sometimes you learn more about a problem by having the abstraction, sometimes you learn more by not having it. Part of being on a growing project is that you get more opportunities to see both sides of a transition and learn from it.

When you have coworkers who clearly have only learned from one half of a timeline and have a great deal to learn about the other, wanting to put the brakes on and have them learn some discipline first is not a crime. Or rather it is, but it shouldn't be. It's like a weird version of Fake it Til You Make it that is verging on fraud. We'll keep up the fakery until someone notices and then we'll get a new job and reset the clock.

I agree, it is quite refreshing to work on an architecture without any caching, although quite slow as well :)

Reply on news.ycombinator.com