Comment by bryanrasmussen | Hacker News Reader

Comment by bryanrasmussen | Hacker News Reader

Parent Full thread

I think the instructor's response in the linked article is a reasonable defense, you don't really know that data is deleted all the way down to the file level. It is just marked as deleted and could be retrieved by someone clever enough to do so. At some point in the future it will be really deleted.

I don't think the GDPR regulatory agencies are operating at a technical level that they would make an argument that it was not a good enough deletion.

Finally I have to ask this part: assuming ES is not your primary database, how does this get around the GDPR issues? If someone wants their data erased you are supposed to erase it from wherever you store data, I suppose this means ES when it indexes a primary store and finds it has deleted data actually deletes it but if it is told to delete something it keeps it around?

When using ES for indexing and not the primary store, you can (and should) periodically fully reindex the data set. You can use a blue / green pattern — create a new index then swap from the old one to the new one. ES supports aliases, making this swapping transparent to the apps using the index. Now you have more options.

If it is easy to delete specific users from the primary database, the deleted users will naturally disappear during the next ES reindex.

Edit: The old index is deleted at the file system level.

If the reindexing occurs daily or weekly, perhaps this will satisfy GDPR.

There are other good reason to not use ES as the primary data store. First, it isn’t entirely reliable. It’s good and I’ve never seen a corruption, but ES and Lucene’s history isn’t as a reliable database. Second, if you want to change how you index, it is a bit easier to do if the source data is outside of ES.

thanks, I wasn't arguing that using ES as primary was good. Just don't necessarily see the GDPR argument as being a reasonable one. Although I've seen some startups using Mongo as primary and have to wonder if there would be that big a difference in using ES at that point (not a Mongo dig as I've kept away from it for various reasons)

Reply on news.ycombinator.com