Hacker News Reader

Comment by kevin_nisbet | Hacker News Reader

Author Here.

Using crypto hashes to anonymize data is one of those mistakes I've seen several times, and wanted to draw some attention to the issue so that hopefully we can all learn from it.

Let me know if you have any questions.

View on HN

So one of the issues here is using an externally visible ID (or a transformation of such) as an internal ID. Why not create a random int64 at account creation time which is invisibly linked to the public username (eg, email address). So now you've got a proper join key, you can restrict access to the map, and it's easy to delete the map entry when the user unsubscribes.

(There can still be good reasons to apply one-way hashing to the random internal UUID, as well: for example, to provide different levels of logs access to different internal users. People who make dashboards get hashed ids, and people who debug logging get raw ids.)

The problem of entropy allowing individual user identification even with all IDs scrubbed is still very real, though, and non-trivial to undertake. One can start by wrapping the query engine with a service which checks that a certain minimum number of people are covered by a given query before returning the results. Or apply differential privacy-type transformations to the output...

This is along the lines of where I was going with the alternative approach, I just simplified it for brevity. :)

In the case of teleport, I think this is a bit more difficult to achieve, because we don't necessarily have our own account database, our common commercial use case is integrated to an identity provider through SAML/OIDC, which I'm not sure would consistently offer a random id per account to use.

While there are many way's we could generate and store the username <-> random id mappings, this adds a certain amount of complexity to get right on a distributed system.

If building a system from scratch with end to end control, I do prefer the random identifier approach.

Then user emails you to ask what personal data of his you have on the server. Now you don't have a connection so you can't find it, but you have it. GDPR non compliance.

If you can't connect it to the user in any way, it's no longer personal information. Expect the data protection agency to compliment you.

> If you can't connect it to the user in any way, it's no longer personal information

Just because you can't connect it doesn't mean nobody else can.

For all practical purposes, a secure, one-way cryptographic hash is irreversible.

I'm thinking of a number between 1 and 100.

It's bcrypt hash is: '$2b$15$qUxzZ5ZF55lMuqiH9GMjQOHkNyee86qd2Vh2kQyF5P3U6JZJx9AEC'

I bet nobody could ever reverse this secure cryptographic hash to figure out what it could be... ;)

I think you need to address converse the examples in the article in order to assert this.

http://www.privacy-regulation.eu/en/r26.htm

... account should be taken of all objective factors, such as the costs of and the amount of time required for identification, taking into consideration the available technology at the time of the processing and technological developments.

The principles of data protection should therefore not apply to anonymous information, namely information which does not relate to an identified or identifiable natural person or to personal data rendered anonymous in such a manner that the data subject is not or no longer identifiable. ...

It's sufficient if one can't reasonably reconnect the data back to the user. It doesn't need to be NSA-proof.

It doesn't say that information cannot be _reasonably_ reconnected, but that you shouldn't be able to reconnect it at all.

I don't know how you have drawn that it shouldn't be NSA-proof from this text if it literally says "in such a manner that the data subject is not or no longer identifiable."

Its in the original link, I may have limited the quote too much:

... To determine whether a natural person is identifiable, account should be taken of all the means reasonably likely to be used ...

Thanks for the quote. Wow. I wonder if this odd definition doesn't render "unidentifiable" to mean "almost certainly identifiable by someone, with a current technique" - since, given enough techniques, most of them will be statistically unusual. I admit it's a start, but mangling semantics that baldly gives me the willies.

The parallel history of cryptography is little more than a history of overconfidence re what counters were thought to be likely, and not. Do we really need to recapitulate that?

It's your mapping, so you can easily gather up everything with the given marker and hand it back to them. You only throw away the key (and delete attached data) if the user deletes their account (and maybe after some additional time elapses, in case they change their mind or were hacked); it's the same process as GDPR per-user encryption key deletion.

If you throw away the key you still have the data but encrypted. There is no guarantee that in 5 years user data could be easily decrypted.

Well, the NSA slurps all Internet traffic, so by that definition, no encrypted communication is possible.

But there's no reason to believe that will be possible either. By that same reasoning it might be possible 'in 5 years' to recover the erased (and overwritten) data from the storage device, so you never can delete anything.

If you use something such as AES 256, which is approved for use to encrypt 'top secret' information by the NSA, and through some miracle it turns out that we can easily decrypt such data in 5 years, then I'm pretty sure you can argue in court that you were following best practices and had no reasonable way of predicting this encryption disaster.

'Key' here refers to the key in the mapping from external to internal userID. The whole point is that (as mentioned in a sibling comment) choosing an internal user ID uniformly at random is equivalent to a one-time pad; it's guaranteed non-decryptable, unless you invent a time machine...

Isn't there a distinction here, though? While they might result in a similar outcome, deletion is different from de-identification.

Isn't this essentially the uuid() function that many databases support natively (even the black sheep MySQL)?

Have you looked into VRFs (verifiable random functions)?

This is essentially a public key crypto version of HMACs. You have a piece of data and a blinding factor, both are committed into a single point on an elliptic curve, and blinding factor is not exposed as-is (as in case with salt for hashed content), but instead committed into a public key (a point).

The resulting point preserves the algebraic structure, which allows you to sign proofs about it, but w/o leaking the blinding factor. You can even sign a proof to a "designated verifier", which makes the proof useful only to a specific party, but anyone else wouldn't be able to trust it as it could be forged by that other party.

No I haven't, this sounds very intriguing though, I'll have to do some research.

Technically a substitution lookup table (like is proposed at the end) is analogous to one-time-pad 'encryption'. In this case the 'pad' is only used within a single (extended time domain) context and is presumably selectively exposed and used only in a contexts where intercepting the context already reveals that data anyway.

Additional security could be added by making a session-unique identifier (not based on user, chronological, or external context data) and only having the master lookup table for user to sessions in an elevated security environment.

Yeah, it sounds a lot like deterministic encryption. Which is OK if your input data is essentially uniform random.

But if it's not, and your adversary knows the distribution of the input data, then the protection level is pretty close to zero.

I think this is the best advice:

“And most importantly, we only collect enough data to fulfill our stated purpose. The fewer data points that we collect, the less opportunity that someone can correlate the data.”

The smaller the domain, the less anonymization works to conceal which user id did an action. However, if you think about it, identity is far more than user id. That is why real anonymity means not storing any unnecessary information from other domains. We have a technique where we use iframes to display a person’s name, friends etc. back to them based on user ids, but the enclosing domain knows only the user ids and their connections.

Please elaborate on whether hashing is a good pseudonimization strategy in context of GDPR guidelines.

I thought the blog post made it pretty clear that it isn't.

The blog post wasnt about

> pseudonimization strategy in context of GDPR guidelines.

i.e. Doing the minimal work to meet the guidelines. Effectiveness is incidental to that goal. Maybe I misunderstood the GP tho.

Reply on news.ycombinator.com