Hacker News Reader

Comment by xtagon | Hacker News Reader

Great article. One thing that wasn't pointed out that might be of interest to someone learning about Event Sourcing is that it introduces some challenges if you are to be compliant with GDPR and similar laws. For example, if your event log is immutable, and you use it as an audit log, then by nature you are not ever deleting data. There are solutions to this (for example, crypto-erasure), but it can be non-trivial to implement.

View on HN

I saw the Akka people talking about this on twitter once, I think they were theorizing that encrypting the data in the log would be sufficient, because then "deleting the key" could be interpreted as the deletion of the record (even though the useless data still exists in the log). But I'm not sure this was ever legally validated?

That is a very interesting approach, however I can imagine that it can be quite a large database of encryption keys.

I'll have to build a small system for this and try it.

Oh, why? Holding one key for every human on earth would fit in 8 TB plus read-mostly replicas. You'd delete their key if they made GDPR removal request and the keyless, encrypted, immutable entries would be entombed in place.

What do you do when the data involves multiple people?

It's unclear (regardless of crypto shredding) what the delete policy should be if the data involves multiple people.

Why would it? Typically you would have one stream per user.

Event: "User A sent Message X to User B". (I think the answer here is again to have PII be in other repositories, and only refer to them by id in the log.)

Good question. Create alias proxies?

That is what I was referring to as "crypto-erasure" (also known as "crypto-shredding"). I'm not sure what counts as legally validated, but some have shared concerns that the encryption you use to do this would need to be future-proof against, say, advancements in quantum computing cracking the encryption down the road even after the key has been thrown out.

if the references don't point to actual data, you don't need this.

- minimum 2 parts. a relay (reference hash) and the cold/true storage portion. you can break the reference hash up and reassemble only for secret holders. which brings us to:

- content-based routing

- there are also deterministic vaults for rolling keys

i'm actually not sure in what high-level situation "crypto-erasure" would work in because being able to re-key a reference means you have complete control. so why would you need to erase the "bad" key when you can just switch the reference?

Eth draft for enabling cold storage relay https://github.com/ethereum/EIPs/blob/master/EIPS/eip-1077.m...

Decentralized ID https://www.w3.org/TR/did-core/

^ both generally use the same concept i described above and solve GDPR "delete" issue. actually, it solves GDPR completely if you can just rely on the DID. "hard delete" is a separate issue, though-- no other way to get around that but to fork/version + replay your store and re-reference anyone who wants to hard delete if you didn't use reference hashes.

Just like password stores. Store salted and hashed password value, vs actual password.

Cleverly applying this strategy to all sensitive datas is the Translucent Database thesis.

Don't stick PII in the events. Write it to a dedicated PII db and reference it in the event. Easy to get rid of the person's data. Just delete from the PII db.

I'm pretty sure you can "soft-delete" for GDPR compliance. So this concern is sort of a non-issue.

Besides that:

1. If you're using an immutable structure, as long as you use references, you can obfuscate data. Blockchains ran into this problem before GDPR requirements and that's essentially all they do.

2. ^ "Update" strategy for event sourcing is the same as above. Essentially a copy of the log or the log slice then a re-indexing to remove/update streams, events, projections, etc. Greg Young talks about the indexing internals (for Event Store) in the 2012 video.

Can you describe more precisely how you're defining "soft-delete" here?

data is still stored somewhere but any routing to the data is disabled and the entity is disassociated.

Could you point towards an authoritative source for this claim?

As a consumer, if I request deletion of my data, I expect it to be actually deleted - not just have a "deleted" flag set.

With soft-deletion, the data is still right there, ready to be abused after a breach.

Soft delete in the sense of removing availability but keeping the data shouldn't be GDPR compliant from what I've seen. Not in regards to right to be forgotten and restrictions on keeping PII.

"soft-delete" is not GDPR compliant.

Your event log should IMO be nominally immutable rather than actually immutable.

You should feel free to take actions such as expunging private or sensitive data as appropriate. Keep the events, but rewrite them to contain only the desired data. Trivial to implement, and simple.

I'd only worry about stuff like crypto-erasure if you physically cannot alter the past, such as if you have a requirement for non-repudiation or some such. Doing it just for technical purity isn't worth the cost :-)

I think cases where you cannot alter the past (or can only do so with difficulty) are fairly common; backups will easily tend to fall in that category.

Backups have the same GDPR concerns regardless of whether you are doing event sourcing or not.

Reply on news.ycombinator.com