Hacker News Reader

Comment by arpa | Hacker News Reader

Okay, this approach is really cool, but... just hear me out: what if... you don't store personally identifiable data in the event store? What if you only store references/ids that point to services which can resolve those ids to data, and that data needs not to be immutable. In the event of erasure request you shred/anonymize that data only, without touching kafka. I mean it's pretty obvious to me, to the point i feel like i'm missing some huge point and grossly misunderstanding the whole problem

View on HN

I think that's exactly what any real system would do. This sort of widespread event encryption and forgetting is crazy. It is a very bad idea to delete events from the immutable event log. There is no telling how downstream consumers will react but it probably won't be good. It's much simpler to only hold references to a user (UserID) and then if you need to forget that a user you simply burn whatever is behind that UserID.

This is actually how the GDPR is designed: it's not about user data it's about personal data, that is data that can be associated or related back to an "identifiable person." The problem is not that you need to delete all the user's data across all your systems, the problem is that you need to break any associations that would allow you to identify a natural person who has asked to be "forgotten." The funny thing is that while some people make lots of noises about GDPR being some huge "burden," the reality is that any architect worth her salt should've been designing systems like this from the very start rather than letting personal data be replicated en mass from one system to another. It is a basic normalization of data. All the GDPR is doing, like most regulations, is requiring businesses follow best practices and not cut corners that might harm end users. The great thing about this in the long run is that this fixes a huge problem with the internet. Today users are reluctant to sign up to a service precisely because they don't want to surrender their data because once it's gone it's gone forever. I suspect users will be more open to trying new services if they can be assured that it's possible to "un-sign up", that is be "forgotten" by a service.

GDPR is troublesome not for requiring something particular which is hard to implement, but because it requires to do something poorly defined, and promising heavy fines for not following the path one can't clearly see. This minefield situation, in fact, makes SMEs much more vulnerable compared to big corporations, and I suppose it's not what most people here would like to see.

> if you need to forget that a user you simply burn whatever is behind that UserID [...] you need to break any associations that would allow you to identify a natural person

The problem is that as you collect more "impersonal" data, the probability that your collective data can still be used to identify someone approaches 100%.

"We don't know their name, but Deleted User 4510 was friends with every single member of the John Doe family except for John Doe himself..."

Surely the implication is that when you 'burn whatever is behind that UserID' you would delete the records of who DeletedUser4510 was friends with (and who was friends with DeletedUser4510).

If all you do is rename someone to DeletedUser4510, you pretty obviously haven't deleted all the data you hold on them.

Right, and what if those connections are burned into an immutable data source through the connection event in your event sourcing?

That is what this article is about, and methods to deal with that.

> you would delete the records of who DeletedUser4510 was friends with

You erase the friend-linkages...

... Then you find that Jane Doe made a post in response to a blanked DeletedUser4510 post, and responses can only be made by friends, so therefore Jane was at some point friends with DeletedUser4510.

So you put in tombstones for all posts and all post-to-post causation links.

... Then you find that the entire Doe family is tagged in a photo by Jebediah Doe, except for one guy, and the comments are every family member and somebody named DeletedUser4510...

Anyway, my point is that it's really easy to get caught in such a fog of relational data that gaps merely change a certainty of identification down to an extremely high probability, and an event-sourced system -- by design -- makes it extra difficult to remove data or to even plan for its removal.

Does the GDPR actually identify what information is considered personal? I haven't followed it at all.

For instance, if someone posted on a message board, is it enough to rename their user to anonymous. Or do you have to go back and delete their user, leaving orphaned records? Or do you have to delete all of their postings, which could leave discussion history in disarray.

What about something like a phone service? Erasing a lines recent history is easy enough, but going back years to delete records from archival systems that weren't designed to handle it could be problematic. For instance, in very large data tables, deletes can be very slow. Call records are often stored in compressed flat files. Which would mean searching through tens of thousands flat files for lines to delete. And some of that data would have been processed through a system like splunk or logstash that isn't particularly friendly to deletes and would require a massive re-indexing operation to flush the necessary records. And some of those systems probably have tiered storage that includes offline, slow to recover archives built with cost assessments that did not account for frequent data removal. (Think about how much it would cost to download 500TB from glacier, decompress it, decrypt it, put it on an active system, remove a few records, re-encrypt, re-compress and re-upload it 4 or 5 times a month). And think of how that cost compares with "we generally need to reference maybe 1gb of data every three or four months".

The law is not defined in a "technical" way. It's definitely not about renaming users or deleting their postings. GDPR is declarative: can you identify someone through the records in your possession, whether database or not, and connect that to any protected information? (their political views, sexual preferences, name and location, economic status, health issues…).

The law is quite broad and fuzzy, which is what makes technical people uneasy:

‘personal data’ means any information relating to an identified or identifiable natural person (‘data subject’); an identifiable natural person is one who can be identified, directly or indirectly, in particular by reference to an identifier such as a name, an identification number, location data, an online identifier or to one or more factors specific to the physical, physiological, genetic, mental, economic, cultural or social identity of that natural person;

It is entirely possible that in your message board example, you've deleted the user, you've deleted their postings, and are still in violation. For one, the person requesting deletion doesn't even have to be your user. Someone else may have posted such information about them on your board.

People misconstrue GDPR to be only about databases and unlinking users' records from tables (mainly because it's the easiest thing to do). But it really is about all and any personal and sensitive information of natural people, full stop.

> For one, the person requesting deletion doesn't even have to be your user.

If what you wrote is true, then organizations are liable for data that someone may, possibly even just theoretically, be able to use to identify someone else.

Given that the law is not 'technical', maybe it'll be interpreted much more leniently than a straightforward reading would lead one to expect.

Yes, it's almost certain it will be interpreted more leniently. At least to start with.

Elizabeth Denham, UK's information commissioner in charge of data protection enforcement, had this to say:

"Having larger fines is useful but I think fundamentally what I'm saying is it's scaremongering to suggest that we're going to be making early examples of organisations that breach the law or that fining a top whack is going to become the norm. Our office will be more lenient on companies that have shown awareness of the GDPR and tried to implement it, when compared to those that haven't made any effort."

In reality, nobody knows how GDPR will pan out exactly (including the authorities).

In that case, individuals are data controllers and not data processors. Everyone posting on an online forum controls the data they post. Adding 'delete post' button fulfils the requirement for a data processor.

> It's much simpler to only hold references to a user (UserID) and then if you need to forget that a user you simply burn whatever is behind that UserID.

But if someone was a user of your service and your services included say photo or video hosting, and you delete their name, their phone number and such, but all of the data you don’t delete still reference a common user ID, then that all of your data can still be related back to an identifiable person.

If instead of holding reference to the user id on the data themselves you have separate table for data ownership that is one step better.

I.e you have all data stored separately and each piece of data has a guid, and you have the user profile separate with a guid and then you have separate tables tying data guid to user guid and when you delete a profile you delete both the user profile and the tables that tie data to that user profile guid.

However I think even that is not enough.

Yes it’s hard to implement full deletion, but you are the one that chose to accept data in the first place. If you can’t implement a system where the data can be deleted you shouldn’t be accepting that data in the first place IMO.

Because even if you delete the references to the data, the individual pieces of data can still be tied back together with data analysis. For example by looking at meta data in pictures, or looking for artifacts that come from lens scratches.

Also, even if you delete references of ownership of an image, probably you aren’t deleting records of comments that other people made on the photo, because if you went to that extent you probably could have just deleted the data proper, so then with access to the image and knowing what other people commented it will likely be possible to determine who originally uploaded that photo.

Not to derail, but rather share: I have worked on a healthcare solution which did just that, every medical record had a guid, and by itself it didn't carry a direct relation to a patient; it was done via an additional table. This was done to ensure data security, ability to use the medical records for scientific purposes (records themselves were anonymous) and to manage access to these records (i.e. patient was the owner of his/her records by default, but could choose to delegate read and modification permissions to e.g. their spouse or physician, or emergency personnel). Come to think of it, it was a pretty awesome system, even though it ran on asp.net (it was some 7 years ago i think). This design was necessary because data privacy regulations for medical records have always been very strict in the EU; come to think of it, people worried about GDPR should take a look at the way medical systems handle data.

One important thing not to forget about: make sure your event log doesn't leak metadata. You may have deleted PII like email or username, but if certain user activity logged under some user id provides sufficient information to identify the user (e.g. a friend tags user on his group photo - this event contains just an id, but allows to link specific person on still available image to that id), what will you do?

That is an interesting point - a photo by itself is personally identifiable information[1] - so you should blur/distort/remove that particular face - or, to stay on the safe side - all faces, just like google maps does to be GDPR compliant, i guess. Jesus, this regulation is such a can of worms...

[1] https://www.quora.com/Is-a-photo-of-a-person-considered-PII-...

If you upload personal identifiable information about me surely I should have the right to get it removed? You already have no right to upload the data in the first place without my explicit permission, so the problem in your scenario is not GDPR compliance unless I gave you permission and if I did I have the right to remove the permission at any point in time already if I so chose. I don't see how GDPR changes anything in that scenario.

The user who posts the photo is a data controller, and it is her obligation under GDPR to remove the photo when requested. All data processors in the chain (the photo app, cache, host, etc.) have to provide the means for the data controller to remove the image. That's my understanding for user-generated content which may or may not contain personal information.

No, the users aren't data controllers, the GDPR does not apply at all to any acts of natural persons in the course of purely personal activity, and GDPR does not apply any obligation to that user (other laws, however, may do that). It does, however, apply to the organization storing that photo - since the default case is that they're forbidden to do that, and as article 6 point 1 (https://gdpr-info.eu/art-6-gdpr/) states, processing is lawful only if one of the listed conditions apply.

It does seem that interpretations handling user-generated content will be tricky, and likely would require additional legislation to clarify that, which is a common scenario for such laws.

https://gdpr-info.eu/art-4-gdpr/

7. ‘controller’ means the natural or legal person, public authority, agency or other body which, alone or jointly with others, determines the purposes and means of the processing of personal data; where the purposes and means of such processing are determined by Union or Member State law, the controller or the specific criteria for its nomination may be provided for by Union or Member State law;

8. ‘processor’ means a natural or legal person, public authority, agency or other body which processes personal data on behalf of the controller;

You are right though that GDPR doesn't apply to non-business individuals, but that doesn't make the image host the data controller. Maybe the user and the host are data controllers 'jointly'? Dropbox is quite clear that at least in their Business and Education accounts, the account owners are data controllers for all their data.

This logic would effectively kill all photo-sharing services. I guess, noone wants to blur people's faces in their Instagram posts,so a regulation following the old Islamic hadith not to depict humans is really terrible perspective.

The whole point of event sourcing is the ability to manage state effectively. When you move state out of the event sourcing model you are losing the state management benefits of event sourcing for that data. Maybe that’s a reasonable trade-off for GDPR, but there is a trae-off.

There are ways to put some state outside the event stream that play nicely with the model.

For example you could store the body in a content addressed store (a key value store whose keys are hashes of the value). This preserves the ideal model of immutable events as you cannot accidentally change the content of the event since that would violate the property of the CAS. But nothing prevents you from deleting an entry in the CAS.

However you still have to find all the records you need to delete!

The trick with forgetting the encryption key let makes this step a O(1) operation.

On the other hand it forces you to decide upfront the granularity of ownership, i.e. if you treat payload deletion as a batch process you can deal with mistakes, change your mind about which data should or should not be deleted. While with the encryption technique, changing your mind would require you to perform a full history replay and reencrypt data in a new stream (and throw away the old one) with the new granularity model.

This is actually a recommended approach within the GDPR legislation itself Article 25: "Pseudonymization"

https://blog.varonis.com/gdpr-requirements-list-in-plain-eng...

Reply on news.ycombinator.com