Hacker News Reader

> Now that storage is infinite and cheap, it's become trendy for engineers to never actually remove anything, but to just set a "deleted" flag in a database somewhere. This makes a lot of tricky things easier, but there are pitfalls.

Also deleting stuff for real for real immediately is really hard.

You need to delete it from the database, the one that's replicating to, caches, online backups AND offline backups. And that's after you've confirmed and reconfirmed from the user that they actually want to delete the data, not just store it in the recycle bin or something idiotic like that.

The easiest way is just to flag as deleted and prune on some kind of schedule. Backups will be overwritten at some point and the data will go away eventually.

Or, though a more technically complex, encrypt every "deletable" unit of data with its own key, have a different backup policy for encrypted data and for keys to it - one which, upon user request, would allow you to quickly purge every trace of relevant keys.

This doesn't work well in practice at any kind of scale -- these types of databases have existed for many years. The key management overhead is considerable, causing integer factor loss of performance for database-y systems. Additionally, it very significantly increases the storage footprint since most techniques for minimizing storage utilization no longer work given this requirement.

It isn't just "technically complex", the limiting factor is your system may be 10x slower and use 10x more storage. The economics of operating these systems is so poor that they are only used in extremely niche environments where the requirements justify the extreme cost and performance limitations.

Soft deletes have been popular since the internet became accessible to the average person. When I was younger I was a huge advocate for soft deleting pretty much everything. It felt like the obvious, easy choice - why not retain the data incase you need to use it for something else down the road? It took years to understand the ramifications of storing that data in a secure way really means, and that often times the reason it gets used later isn't something you intended it to be used for when you designed the data structure.

I don't think that way anymore and I truly believe hard deletes should be the default unless you have a very compelling reason to soft delete (and you should usually only soft delete with some guarantee of future deletion via a publicly accessible data retention/erasure policy)

It's not rocket science - encrypt with a one time symmetric key, encrypt the key with a public key, store blob and key marked as deleted. Store the corresponding private asymmetric key on a hardware token to be used when you decide you need the deleted data. That way no db hack exposes any usable data.

DB hacks aren't even close to being the only (or worst) threat. Encryption means nothing in a company or bankruptcy buyout where the purchaser may have a completely different set of ethics than yours.

That is all good and well against simple actors, but does not defend against the company being bought out and the new owner(s) deciding to decrypt and sell the entire database to some shady data broker. Or against the FBI/your local equivalent coming over for a friendly visit and confiscating both the hardware token and the database.

>This makes a lot of tricky things easier, but there are pitfalls.

Some people are calling excessive data "toxic asset", due to the future risks of it causing unexpected breaches of privacy, damage to reputation, or other losses.

We'll certainly develop standard technical means of handling that in due time, quite possibly ones based on cryptography. A common scenario at present: instead of trawling all backups & distributed stores to laboriously expunge every bit of deleted data, you could delete the relevant encryption keys - a small piece of data, a known quantity - rendering the encrypted data inert. This is particularly suited for systems that use WORM approach to backing up & distributing data.

> just set a "deleted" flag in a database somewhere

this has been a thing since the 1980s, at least. It's not new, and it's often important for auditing purposes and because mistakes happen.

What's new is the ease with which databases can be accessed and hacked over a global public network.

> Or an errant database migration isn't going to transfer it properly. Or the database will be leaked, and all of the "deleted" content will still be there

Backups are a thing, you know. When you issue a delete command to your database it's not reaching back to time immemorial through the archives and ripping out those records. If we're playing what-ifs, then backup databases can just as easily be leaked.

This is exactly what happened with the AshleyMadison hack. In fact, users explicitly paid for a "delete my account permanently" action.. yet it was just that, a "deleted" flag added to the DB which eventually got leaked.

Just look at Parler. That's the same thing which happened there.

It's not trendy, it's how I've been doing databases my whole life. You don't let users actually delete stuff from your database. That's crazy.

How would you handle a user ceasing their account and asking you to remove data you no longer need to provide them with service?

Under GDPR you'd be required to really delete that. Obviously GDPR laws don't hold everywhere but are you saying you'd just hold the user's data forever or that you'd have a later clean up process or something?

You wouldn't. GDPR didn't exist. That was my point. We've been doing this for ages and only recently had to worry about actually deleting stuff.

This is why I've never advocated "deleting" Facebook accounts etc. If you "delete" it you're just giving them another piece of information about you. Namely that you want your account to be deleted.

The GDPR has not yet been shown to have any teeth, so I assume nothing has really changed in this respect.

And, yes, holding users' data forever is normal. Storage cost has only decreased relative to the size of these databases. I've worked in places with customer data going back decades.