Tell HN: Royal Mail Data Leak | Hacker News Reader

Tell HN: Royal Mail Data Leak

85 pointspmx33 comments

Royal Mail (the UK's postal service) has a product called click and drop that allows businesses to pay for and print shipping labels online. It has some value-add features like order-syncing to make buying labels easier. Today when loading pages on click and drop it will show you details from some random account each page load. We saw details of other businesses orders and customer addresses before we logged out and called them about it. We asked another business if they noticed the same and they confirmed that they had.

View on HN

A similar event occured on the Steam Store in 2015 due to a caching problem: https://arstechnica.com/gaming/2015/12/valve-explains-ddos-i...

It seems like good practice is to check data retrieved from the cache is what is expected, e.g. the user id from the cache matches the logged-in user id.

Unfortunately, most devs don't think there is ever a need to check that until it fails.

I always put the user id in the cache name.

Something like: user_profile_bio.[user_id]

If that returns the wrong bio from Redis, then something is broken in Redis, no? Or are you suggesting that I create some sort of format that duplicates the user id in the data itself?

Works great until the Vary configuration on your CDN accidentally drops that.

Sorry, drops what? The cache name or part of it?

Oh sorry. The configuration which tells your CDN what is unique and uncacheable about a given page. If that goes wrong, you end up with situations like this.

Should you add a user_id in the data and check it's the one you want? Yes, I think that is exactly what they're saying, relying on the caching mechanism to handle your data protection sounds like it may be one of the steps involved in the bug Royal Mail is facing.

Undoubtedly you are sure this won't ever go wrong, but all it takes is someone else to cache something at user_profile_bio.[customer_id] and you've got a problem (at best it won't work, at worst you'll get the wrong user profile).

Yes. Adding a small amount of redundant data to your cache as a safety check check is important.

Imagine instead of a user profile at Royal Mail this was patient data in a hospital. Getting this information wrong could endanger someone's life.

And mistakes with private data that isn't safety critical can still cost a hefty fine from a government data protection office, or bad publicity and damage to your reputation.

I dunno, I guess the same could be said about your database then, no?

If you're doing:

    SELECT * FROM serious_money_table WHERE user_id = :user_id;

How is it all that different with Redis? Someone can mistakenly send in a :customer_id when writing records, but I don't really know anyone out there that also stores user_id into every table they query user_id from.

If you are querying for user_id by definition the table contains user_id. I think what they are suggesting is checking that the data returned from the query has the correct user_id. No one would do that because if you can't trust your database to answer queries correctly you have more problems than just that. My guess with the Royal Mail is that it is probably something dumb like using a field in (Java) servlet to hold the data which is then shared across all sessions. I have seen that multiple times in my career. People don't notice when testing as there is only one session at a time.

I'll be honest I don't have much experience with Redis specifically, I do however think that the caching layer is likely the wrong place to handle data security. It seems too easy for someone to accidentally circumvent the cache and thereby the security. I do think that organising the cache to catch problems early is a good idea though.

The thing is that a function like

    def get_profile(user_id):
        profile = fetch_profile(user_id)
        assert profile.user_id == user_id
        return user_id

makes it a lot harder to fuck up, doesn't matter if you swap some ids somewhere (i.e. fetch_profile actually uses the customer id, not the user_id), or if the cache fucks up, or if you accidentally wrote 'SELECT * FROM user_profile WHERE user_id = user_id'.

Works fine until user_id is null for whatever reason, or you hit the length limit on your cache key and start losing the end of it, or until your cache server decides to just start serving up the same content for every key, because y’know, software (had this happen with memcached a long time back due to a tcp timing bug and a corrupted packet) and then you’re in the soup.

One time years ago I visited Youtube and for a few minutes I was logged in as a different user (some guy from North Europe). I could look at their Google profile, etc.

It was crazy. I don't think I've ever told anyone (how would it come up?) but this reminded me of it.

Ouch. I've seen this happen (luckily never in production) when caches doesn't get keyed properly.

Klarna (pay-later lender) had a similar production issue a little while back: https://www.klarna.com/us/blog/detailed-incident-report-inco...

Now currently down for planned maintenance.

EDIT: https://clickanddrop.statuspage.io/incidents/8cd3bf2qyz5h

The person I spoke to called me back and said "They've turned it all off". I wouldn't want to be working in their devops team today!

Problem with their cache (redis / elasticsearch / etc.)

Happens even to the best companies.

Interesting how many people go straight to the cache - seems like we've all been there when we're accidentally overwriting keys with the wrong data!

Experience is the best teacher I guess, although we all seem to be doomed to make the same mistakes.

Heck, at least it wasn't DNS... This time. DAMN YOU DNS!

"There are 2 hard problems in computer science: cache invalidation, naming things, and off-by-1 errors." -- Leon Bambrick

> Happens even to the best companies.

Er...no. This is an inexcusable screwup.

Anyone else believe royal mail parcel details are getting siphoned off at some point and sold to scammers.

Every time I get a parcel through them I get a phishing sms about the parcel.

There are a lot of ecommerce integrations with Royal Mail that have access to order data, it's more likely it's one of these 3rd party services selling off the details than it being Royal Mail

When I created Amazon account I had some French delivery address and some card. I tried to use it, but they asked for CVV. So I deleted it. And added new one.

The account was new and I never used Amazon before.

I did received the book thou.

Sounds like caching issues resulting in a leak, not an explicit breach.

It's not a hack, but it's still a breach.

A data breach just means that data is accessed by unauthorized parties. Accidental data breaches are not only possible, but common. If a company accidentally emails a group of customers with CC instead of BCC that's technically a data breach, although in most circumstances a low-impact one.

If it's actionable by the ICO, it isn't a low impact one either.

Breach used in this case is really spin. Calling it what it is, a "inadvertent disclosure" puts the blame where it belongs.

We call it a breach because the original sense of the word refers to an opening, tear or rupture. In that sense, this is exactly the correct term. Who or what caused the opening is irrelevant.

Hey man I'm just trying to get done work, breach my front door and than breach a few cold ones.

If a bank left $1b of cash at a bus stop and somebody takes it, it's not an explicit bank robbery, but it's still a theft.

This however is worse, as the property they were negligent with wasn't their own property. It's the bank taking your safe deposit contents and leaving it at the bus stop.

Ok, we've made the title say leak instead. Thanks!

Reply on news.ycombinator.com