Hacker News Reader

Hi. I also work at this scale. It doesn't take a year to delete the data. Be upfront with your users:

'Your delete request is being processed. No one will be able to see this item though it may take up to three days for the data to be completely removed from our servers'

This is not a technical challenge, it's an issue of priorities and Facebook doesn't prioritize privacy. They were able to release a TikTok clone in a few months, but can't solve deleting data?

Deletion is very much a technical/organizational challenge.

Imagine some engineer writing some feature you've never heard of and they decide they need data in a different format. They write a job to do this.

Now imagine that job somehow has a bug which prevents the deletion system from understanding that copy exists. That bug could be non-obvious to find.

>It doesn't take a year to delete the data

It does if you have cold backups that take a year to cycle out. They're often offsite, compressed, and incrementally hashed so finding individual items and removing them is really hard -- you're much better off just waiting for them to expire, and a year isn't an unreasonable amount of time.

The article specifically states this data was downloaded using the "Download Your Information tool on Instagram". The undeleted data wasn't from a cold backup or any backup, it was still on production systems.

If this is a GDPR "right to access" request, which most "download your information" tools are, then backups are included. See https://law.stackexchange.com/questions/27625/gdpr-complianc...

Doesn't that link show that the how backups related to GDPR is still up for debate? The accepted answer that says they are included has 3 upvotes and there is an answer that says no they aren't included that has 2 upvotes. Either way, the legal requirement is a separate issue from what Instagram is actually doing. If that download tool is automated, I am rather confident in saying that it isn't combing through year old backups to get data.

Ever wonder why "download my data" usually takes a few hours/days on most services before you receive the download link in your email? I promise you compiling the production data doesn't take that long.

Backups are covered under GDPR, although when a user requests erasure you can say "your data will be rotated out in X months/years". Not sure how this applies to access, but I assume it's similar: https://ico.org.uk/for-organisations/guide-to-data-protectio...

I‘m not sure whether (if any) you habe cold backups at that scale. I imagine cost of the backup would be way to high.

That's what, being upfront about something makes users to trust them but it's Facebook we're talking about.

Whenever I see someone write a comment like this, I want him to understand, that don't say deleted, if it's not deleted.

Also, if it not works at scale, don't tell your users that it works and pretend it works as you communicated.

I find really bothering, when engineers trying to hide behind the "it's a very complicated process in the background, so we just say we have done something, when the truth is, it is in progress. It's not a user error, when you see the word "deleted" you assume, your picture is gone, it's your error, if you state it's gone, while it is not.

Developers are also falling into this trap regularly with distributed systems. MongoDB famously claimed to be "eventually consistent" but in reality failed to deliver on this promise. The MongoDB authors then tried to weasel their way out of it the same way you are describing and many developers rightfully called them out on it.

I agree with you completely, no one should be hiding behind "It's a really complicated process". They need to actually sit down and come up with a concrete plan on how to tackle the problem then communicate that we have a plan in place its been added to the roadmap and we are going to tackle it in Q3 20XX.

Bingo! Its the same as saying we take your username and password and let you login. Do you or do you not? Its a boolean feature that either does or doesn't do what you say it does. Deletion can be complicated and if it is provide that indication to your users. Software is about setting the correct expectations, otherwise we leave users to interpret written language which we all know leaves room for misinterpretation due to all sorts of complexities with the way humans communicate with one another.

This rationalization isn't acceptable to me. They have gladly set up the infrastructure to handle a high volume of new information and it should go both ways.

should See RFC 2119, "should" usually denotes something as "optional" But really, almost zero companies build software around data in such a way that the data could and will disappear at any moment.

Their comment wasn't an RFC. Right to be forgotten and GDPR are not suggestions.

Regardless, in practice, billion dollar companies have the resources to ensure 'delete' isn't actually "partially hide for years/forever". With proper surrogate IDs they could even preserve a minimal amount of meta such as tombstone IDs to ensure all first and third party systems are in compliance.

Woe is me. My poor little (looks at notes) trillon dollar company can't cope! Instragram seems to be able to cope with 100000 new photos per second - that they can do. Across geographies and data centres. Ever seen a user facing error due to a newly uploaded photo?

No one expects you to just rm -rf. They just expect you to expect the same effort to remove the file that they put in to create the file. It's not like the file got propogated to 10 different data centres by accident. There was a decision that an uploda needed to be everywhere in 5 seocnds and a deletion was an advisory hint that could be ignored for a year.

Google has an internal standard for how long it takes to wipe out user data after a deletion request, and compliance with it is taken very seriously. It's a lot less than 1 year.

I think this points to (one of) the significant flaws in the cult of the "minimal viable product". By eliding up-front design tasks, you may implement something in which complying with the law is impossible. Data retention compliance has to be designed in from the beginning, or it might be technically infeasible to add it later.

I think this is why you need a very senior engineer overseeing your MVP.

Before MVPs were championed, companies would create a full-fledged project management suite with 50 database models and release it to crickets.

An MBA and a junior engineer define an MVP as “You can log in, and add todo lists and check them off.”

A senior engineer defines that same MVP as “You can log in, add todo lists, check them off, delete your account, and if the server crashes or we’re hacked, we have backups for 30 days.”

Or in other words with an MVP you still need to complete the 80% of the iceberg the MBA and junior engineer can’t see, but you don’t build the whole iceberg at once.

...and I do not believe for one second that they have never discovered a copy of data lying around somewhere after they thought it was deleted. The choice is to admit it, or lie about it.

It has actually happened and they did come clean about it. Some data was discovered in log files that shouldn't have been there. They owed up and fixed it pronto even if the chances that the data was leaked outside of Google were nil. There is plenty wrong with Google, but that isn't one of those.

If you can insert... for sure you can delete? It doesn't take days for your picture to show up.

Sure, eventual consistency and replication and distributed storage is hard.

Alternative approach: encrypt at rest, delete encryption key. It would require extra resources for sure, but if privacy was a concern... ;-) (and then have a reasonable expiry on your CDN if you use one).

(of course then we can argue that deleting the encryption key may be difficult too, for the same reasons)

The cost of insert and delete is massively asymmetrical. This is an intentional and fundamental architectural decision in all of our data infrastructure and built into the data structures and algorithms that are used. Not only does “delete optimized” data infrastructure not exist, in many cases we don’t have good computer science for how you would even design such a thing while having good performance for all the other operations.

People that suggest using encryption and throwing away the key have not thought through the implications of that approach. It scales very poorly and therefore is not suitable for most practical systems. There are good reasons “obvious” solutions like this are not used.

I would say a system that can't effectively delete also scales poorly. No?

It depends on the definition of “delete”. Removing a record from a data model scales just fine, and is how it has always been defined in database systems. Making that record physically unrecoverable from any hardware in the broader system is extraordinarily expensive for fundamental technical reasons. Traditionally “delete” has always meant the former in all systems, and physical deletion is deferred to a point in the future when either the an opportunity arises to do it inexpensively or the cost becomes acceptable. There is a value in recovering the resources consumed by the deleted record but it comes at a very high cost.

If you change the definition of “delete” to mean unrecoverable physical deletion, then sure, it scales poorly. But it is a bit like redefining “fast car” to “can travel faster than Mach 5” — technically a valid as a redefinition while completely ignoring the engineering realities of what a car can do.

It's not a redefinition at all!

Actual deletion is what Facebook told us they did; that's what everyone assumed when they said they were complying with GDPR. This is like advertising a car as 'faster than Mach 5', and then when people call you out for lying, saying, "well, 'faster' is a relative term."

This kind of crap is exactly why people don't trust Facebook. It's not because people are paranoid, it's because Facebook systematically creates expectations in their advertising and public releases that they're acting responsibly, and then acts like they're the victim of unfortunate circumstances and misunderstanding whenever they get called out.

If three weeks ago a tech journalist had written an article saying, "Facebook will fully delete your data when you ask", no one at Facebook would have been reaching out to that journalist saying, "Oh, in the interest of preventing misunderstanding, we don't actually delete the data, we just mark it to be ignored." But now that they've been caught, now it's just a big misunderstanding by people who don't understand database architecture.

When companies tell the public that they're doing something, it is reasonable for the general public to assume that they're referring to the commonly understood definitions of the words they use.

You’re right. Maybe the technical definition means altering a record to say DELETE=TRUE, but the widely used definition means unrecoverable, and that’s what matters.

I find it very interesting that given a chance to hover up a billion records we see no technical challenge at all but when asked to get rid of them suddenly you're demanding unobtanium.

If you can't reliably get rid of the data you administer, don't collect it in the first place.

Just because it's hard doesn't make it optional. If you operate at the scale of Instagram, yes, it becomes very hard, but the companies typically have both the revenues and expert staffing to do it.

And if they don't, they're mismanaged and need to allocate more resources to it, because it's illegal not to, for good reason.

I’m hesitant to think they just couldn’t figure out how to make it work. Giving that company the benefit if the doubt seems like a bad idea.

> you literally can't just delete data

You literally can.

You can't remove data that you don't control though e.g. Removing embarrassing pictures from the internet. However if you control the server you control the data. I can appreciate that it might not be as simple as a unix command but it is a single letter in CRUD. It's fundamental.

I don't think people understand the vastness of resources that FB and Instagram have at their disposal. This is not a particularly hard problem. Yes it's not a simple SQL query. But they do solve it -- in fact this was a bug that they have already fixed.

Whenever someone writes a comment like this complaining that something is too hard I always want to remind them that that isn’t an excuse.

They've got 30 days to delete data, there's no excuse not to manage it within 30 days.

Also, you're saying they intentionally built a software stack without even thinking about how to obey the law (the 30-day deletion requirement existed even before the GDPR, since 1996 even, just with lower fines)

I hope in a few years this case will be taught in school just like the Therac case is being taught right now.

It's not that simple, some of these companies have tech stacks that are run-a-way trains written by people who had good intentions but wrote their code and built their infrastructure in a way that was meant for 1/50th of the traffic they currently have. I don't know anything about instagram's infrastructure but what I can tell you for certain is that every one of these companies has legacy infrastructure and code that keeps them from being able to meet all legal requirements. You can't just make a blanket statement that they don't have an excuse.

> I don't know anything about instagram's infrastructure but what I can tell you for certain is that every one of these companies has legacy infrastructure

So you don't know anything about Instagram but at the same time know for certain that they have legacy infrastructure that keeps them from meeting legal requirements? How about we don't give them a pass just because it requires a bit of work to comply with laws and regulations.

Maybe its just me but I have yet to work for a company that didnt have legacy infrastructure that was hold them back in some way. Even the startups I've worked for had to make odd business decisions in order to take on new customers. Those odd one-offs and business decisions eventually led to sprawling legacy infrastructure. If there is an infrastructure Utopia I have yet to work there or hear of it.

If they don't want to comply with laws then they shouldn't be operating.

That really isn't the user's problem.

Laws and rules need to be followed no matter what.

It's the same with GDPR.

And this is the attitude that causes businesses to tell lies about everything they are doing. If you want transparency with regard to how you're data is being handled you need to ease up a bit and realize there are biological limitations to how fast people can make changes. Most of these companies have people who want to fix these problems. They just have hurdles you don't know about, some of them are internal politics and others are technical.

These companies are shipping new features every day. If they have the bandwidth to work on those projects then the failure to meet legal requirements is an issue of prioritization, leaving no excuse.

Then they have to stop adding new features, and focus their countless developers on implementing a way to meet these legal requirements.

And the fines for not doing that need to be big enough to motivate them to do that.

This. The amount of time quoted in the article in excessive but it's also sparse on details (maybe his data request also sent him backups related to him? a year is reasonable in this case).

Deletion is pretty much the hardest thing to do in data management, way harder than inserting or retrieving data. I know GDPR says you need to be able to delete a user's data, but how reasonable is it to trawl through your offsite backups to find their items and remove them?