"developer gets a great idea - let's push an update to the API as a GET request so we can cache this on the CDN... forgetting that the JWT token is potentially returned in the call. Now, whoever makes the call first gets their JWT token stored for everyone else to load instead when the API call is made."
Ta-da, Klarna.
When you test features like this or caching a response with a JWT it can be very easy to default to the happy path or ignore the impact of a large volume of concurrent users.
Nope. That definitely wasn't an engineer.
We immediately started getting reports of random products appearing in our customers' shopping carts, as people's sessions got merged with random strangers.
However, one common source for this kind of bugs is to ”cache any URL ending .pdf as a static file” and then you are in fact serving logged in PDFs like customer invoices that come with the session cookie.
I think CloudFlare used to come with a default rule to treat .pdf as a static content. The responses were cached when you hit their ”cache the good stuff” checkbox.
Manhandled in-house though...
Such is the cloud software... :) Cloud, besides APIs, i.e managing hardware at scale was not really what they did.
They did roll 1000s of vms per week through it in ci/cd flows though.
As such it did what it was supposed to do - docker/containers was not a thing at that point in time, and I remember thinking it was pretty awesome.
To many nifty engineers, with long fingers, for their own good though. You need to be strict with automations if you want to keep something like that running reliably over time.
But... API's really shouldn't be cached? At least not at the CDN level. The risk of serving up stale dashboard data alone makes users go ????... and we definitely don't want - not even mentioning the problem here, that's crazy.
Cache invalidation is always a very tricky affair. It can work for a while but as complexity grows it gets very hard to maintain and debug. It's very much a "here be dragons" situation and you have to go into it with your guard up.
I was at a small startup that had a quick and dirty contractor built API. It worked, but for our largest customers, 99th percentile latency started going over the API gateway timeout. The quick and dirty hack on top of it was aggressive caching with too-clever invalidation logic. It worked until new features were added and then it started failing dramatically and unpredictably. The bugs were an absolute nightmare. We ended up spending almost a year cleaning up the data model, sharding things by customer, and fixing a bunch of N+1 queries, all so that we could get rid of our API cache layer and kill the bugs for good.
A couple of years back, I was making https://lifeboxhq.com which involved users uploading quite a bit of content. I was happily testing security with some url resource enumeration and for some reason, I could non-deterministically access user uploads via url, even on accounts I didn't own. I spent several days looking at my Flask code, javascript, etc. to debug....
I knew it wasn't my code, but I was getting more and more frustrated, then I remembered I set up Cloudflare....
Remember to exclude certain routes from Cloudflare if you want to avoid arbitrary user content from being cached without authentication.