Post Mortem: Incorrect Cache Configuration Exposes Personal Information
klarna.com
klarna.com
At the mild end, you could get stale data. Deposit some money and the cache doesn't reflect making you wonder whether the deposit happened or not.
On the worst end, you get this - all users get exposed the most recently requested data.
All this to save... a few kB?
I've run some eCommerce sites before, we were PCI Tier 1, roughly 1% of web traffic: but nowhere _near_ the level of security Klarna should have; we had the CDN's activated only on subdomains... always.
Usually static.<site>.com or whatever..
This has the added benefit of not sending cookies to the CDN, because that can affect latency and is also a bit less secure because a static image/css/js does not need the cookie. (and bandwidth, before we were charged per byte (and that's egress now anyway in most cases, god I'm old.)).
If I had to venture a guess, they're probably scared of getting slammed with a DDoS so they "guard" everything behind the CDN, it's not about caching.
We got slammed with DDoS's too so we had to have systems in place to push BGP routes via verisign to "scrub" the traffic... we didn't even have that many "credits" and it cost millions...
If they separated app/cdn/web with different SSL certificates, something like this wouldn't be possible even when developers eff up.
Passing dynamic non-cached requests through a CDN isn’t about saving a few kB for the server (by definition it still has to generate it). It’s about improving end-user performance through things like having edge termination of TLS.
So it’s not so easy to have such a blanket statement as no API requests should go through a CDN. While it’s all too easy to inadvertently turn on the caching of requests.
Err, wouldn't the CDN itself still need to do the same TLS handsake with the end server?
This creates a complete disconnection between whatever TLS handshakes are happening between the client and my CDN (Cloudflare) and whatever's going on locally between the tunnel daemon and my local web server process.
For example you can choose to run these tunnels without HTTPS at all on your local webserver (bearing in mind that you have no inbound firewall ports open to the webserver box - no traffic leaves that box unencrypted), the daemon directly proxies your local-only HTTP port on the machine over an encrypted tunnel to Cloudflare and via whatever flavour of Cloudflare TLS you like to your end-users.
Another reason a no-caching CDN can help is when that CDN has a higher quality connection to the backing service than the user has. A lot of eyeball networks out there might have terrible transit connectivity to the global Internet during peak hours (because that cost money), but generally have high quality settlement-free peering with CDNs. By paying a CDN, you're effectively paying them to provide better Internet connectivity to your users.
In fact, I think that the performance gain is not worth the extra complexity for most sites out there. The recent fastly outage, klarna outage, ... all show that CDNs are no silver bullet.
All infrastructure ran in us-east-1. A considerable percentage (but not the majority) of their customers were based in Australia, and quite a few of them were complaining about sluggish performance. HTTP/1.1 + TLS meant connection handshake alone took over 5 seconds! Before the first useful bit of content was sent. Add in the additional pain of a web framework that was still splitting assets across multiple asset hosts (to compensate for old browser “feature” that limited the number of concurrent requests to a single domain”). Over 30 seconds of connection setup, some of it in parallels, almost all of it blocking page render because it was for HTML or dependent scripts.
Add in a CDN (no caching) and edge termination meant single-digit millisecond handshakes to a POP in the closest major city. Also had the ability enable HTTP/2 at the edge without needing to change anything else which meant improved connection pooling for clients and less connection setup overhead. Backhaul connection reuse also has the benefit of pooling all requests via a specific POP rather than each individual client. We got rid of the asset host thing while we were at it. Just a CDN, no caching, and a greater than ~30000ms reduction in request times for anyone far from us-east-1!
That is simply not correct.
The entire reason why Fastly is so popular is because it lets you cache the API using cache-bust on write. Between the request collapsing, shielding, and a near immediate cache invalidation caching API allows one to accept enormous spikes of traffic that normally would bring down the origins.
What is Klarna _for_? What does Klarna do that others don't? They brand themselves as "We enable you to buy hip shit you don't have the money for [and will become irrelevant before you payed off your debt]". Almost literally [0]
Is it more of the same but dressed pretty or am I missing something?
That said, sometimes a common thing done up well will be successful. Looking at you, Slack.
[0] I should say I can't remember for sure if this was from his time at Klarna or somewhere else, but it was at least illustrative of the kind of people and the culture there.
There are options for paying first after you receive the goods, and this is all handled by klarna so its a huge win for the sellers.
I don't know how they operate in US but in EU-land they have been pretty awesome. Probably 80% of online stores in Scandinavia use them.
On the flip side, they may have been doing something super awesome for the vendors, since it's so widespread that it's used on almost any web site.
I found my experiences more in line with yours, payment is clunky, and seems dubious.
So Klarna delays having to authenticate with the bank which issued your credit card from the time of purchase, to the time of paying your Klarna "credit card bill".
They also delay sending your invoice until the item is actually shipped, and you then have a 14 day due date. This means returns and refunds have no money exchanged from any of your own accounts, so you don't have to worry about how long it takes for the refund to enter your credit card.
Edit: They also remember you address based on your email and name, across any vendors, so check out is really quick even if it is the first time you shop at a specific site. This is of course great for smaller vendors.
The checkout is super quick only to be frustrated with the payment process later. And since it's Klarna that comes after you to collect payment it can eventually backfire on the business that you ordered from as well.
Also in my country, in Europe, Klarna is mentioned more and more in news articles about consumer debt and predatory lending, so I think their market share among people who keep credit card balances is increasing. They have stopped or kept pending unusually large purchases by me pending a credit check, but they could probably do a lot more to not get a reputation as lumped together with the most predatory consumer loan banks.
The shop you ordered from does not know or care how the transaction is done between you and Klarna.
Klarna: Our app is configured specifically to avoid inadvertent caching through the use of cache control directives found in the HTTP standard (no-cache, no-store, must-revalidate). However, the CDN configuration can at times override the app configuration.
Zulip: Zulip separately marks its responses as non-cacheable, both to prevent private data from being sent to the wrong user, and also to prevent stale data from being sent even to the right user. Indicating that a response is not cacheable is the purpose of Zulip’s Cache-Control header: max-age=0 [...] no-cache [...] no-store [...] must-revalidate [...] private [...] CloudFront usually respects the Cache-Control and Expires headers, but with one critical exception when receiving multiple similar requests at roughly the same time
It looks like Klarna's main website uses CloudFront, so I bet this was the exact same thing.
Luckily, I haven't used it. I never liked that one need to provide his bank login data to Klarna / Sofort to use it. How is that even legal?
EDIT: I haven't heard about Klarna outage, Klarna is well known.
Either PayPal or CC issuer provide at least some buyers' protection, while given your bank credentials to Klarna is normally against ToC of your bank.
CCs are not a great mechanism and I wished we had better bank services and safer integrations to process payments, but that's what we have. I check my bank statement regularly and I know I can get money back if fraud happens. If I don't trust a website I used a prepaid with the exact amount on.
That's the whole point of klarna (who also do CC processing so might have used them unknowingly): you don't need to trust Johnny's Little Online Store handling your CC, since the payment is handled by klarna.
But in practice it's either invoice, PayPal or CC. Even if Klarna process that payment in the background, I can still object to my CC company and let them settle it, instead of having to deal with them.
Also, Klarna is a major invoice and CC processor. You probably have already used them unknowingly.
I am parroting other HN comments here, but the reason this is necessary is because banks do not provide a public (or private) API except between other banks. Therefore the only avenue for integration would be directly accessing the account.
It's worse than that. Several banks tried to enforce in their terms and conditions that sharing your bank login data would be a breach of contract or at least absolve the bank of any liability. This was deemed in violation of antitrust laws for some reason, so now banks have to actively allow this which is just brain-dead IMHO.
The only upside of this is that we now have PSD2 now anyway, so the damage that can be done by someone who has your login credentials is limited.
My best guess at what that "some reason" may be is that it breaks shoddily built integrations like [Teller](https://teller.io/), which, instead of using schemes like Open Banking where available, simply rely on credential sharing.
Is anyone else shocked by this? Are code reviews worthless? An engineer made a change reviewed by 3 other people. It seems like _someone_ would notice.
> Code reviews in my prior experience solved nothing
I find that assertion suspicious, unless the culture was "push the approval button without even reading because ship,ship,ship"
Let me fix for it.
"Detailed incident report: No One At Klarna Understands That Disabling CDN Cache Protections for Personal Information is Bad"
This post attempts to blame their CDN vendor.
“All these reviewers couldn’t spot it.”
In my limited experience working with CDNs wouldn't you just cache the responses of unique URLs and have some sort of cookie check at the edge before serving it.
So my own app would request something like /api/account?id=123 with my own id in there.
How would you end up getting other people's data in your app if your app only calls that unique URL?
From what I understand what happened with this outage, the CDN would still cache /api/account?id=123, and someone with account ID 234 could access it by altering the URL to retrieve the cached version, if account 123 has used the app recently.
That's because a CDN has (usually) no concept of authorization/authentication and can't make decisions that /api/account?id=123 shouldn't be served to someone other than the owner of account 123.
It would be less catastrophic (at least from a PR point of view) because people wouldn't get immediately served others' accounts, but you'd be vulnerable to attack.
But what GP seems to be asking about is: “Would having your app always encode a user ID into the endpoint have helped?”.
I originally thought they were trying to cache account data responses and so wondered why they wouldn't just use unique query parameters in that case. Definitely risky business though.
Out of interest, it looks like Cloudflare offers some sort of token authentication to authenticate at the edge: https://blog.cloudflare.com/token-authentication-for-cached-...
Edit: If the other commenter is correct, then it's less bad than I imagined. Or rather it would at least only be triggered, seemingly, if someone deliberately and maliciously requested something that didn't 'belong' to them.
I guess I didn't think of that originally because I thought if you wanted to cache some kind of response data like this, why on earth would you use a generic URL? But perhaps they probably didn't intend to cache this endpoint.
> HINT: One of many reasons you should be using Cache-Control and not Expires for your caching configuration is the ability to use directives like private. The private directive tells Fastly that this response is good for only one user, the one that originally triggered the request. Anyone else that got tagged onto the queue must now be dequeued and their requests processed individually.
> Responses not marked private, may be used to satisfy queued requests. Even a response with max-age=0 or the equivalent no-cache can be used to satisfy all the waiting clients, and the next request for the object will once again be a MISS. If your reason for giving a response a zero cache lifetime is that it contains content intended for a single user only, then ensure it is correctly marked private to avoid having this content sent to more than one user.