Hacker News Reader

Comment by rafinha | Hacker News Reader

I'm not sure such thing exists: "large companies with lots of medical data". Medical data is often confidential and belong to hospitals.

View on HN

I used to work for a medical lab and worked on a lot of projects that involved aggregating and cleaning medical data to sell. Often it would just go to pharma companies so they could target the best places to sell their drugs.

Anyway, it's completely legal. You just have to scrub the data pretty thoroughly before you sell it.

I think health insurers know every little medical detail about you from every interaction with the health care system you have that they process a claim for.

The dataset spanning all of them is likely to be in the tens or hundreds of TB range, if not PB.

I've worked in a large company that has "lots of medical data" which we used for analytics purposes. What we would do is "Anonymize" the data. Unfortunately that's a pretty herculean task.

What's involved? It sounds fun.

So the naive method is to just remove any information such as name or birth date. However the medical data is still identifiable. So we need to alter that too. However, we did analytics, so altering it might ruin the results. So we needed to design ways to scramble the data so that it wasn't identifiable, but that in aggregate the results stayed the same.

Did you use something like Differential Privacy?

I would be very interested to know if anyone is using that in practice. It seems like a promising concept.

Apparently, DP has some detractors. I was told by my signal processing professor that differential privacy wasn't really a solution for privacy preserving data analysis. He said something along these lines: "if I know something about the underlying data distribution (Gaussian, etc.), it is possible to wash out the randomness."

Now, I don't understand DP well enough and information theory/signal processing still seems a bit like "dragons be here" to me. But, I want to take a stab at trying to reason why he said that.

For example, take randomized response (the only DP technique I understand). That is vulnerable to a longitudinal attack: a person can query repeatedly to wash out the randomness. If you think about it, isn't it the almost the inverse of a repetition code (error correction)? There, you're trying to use redundancy (repetition) to remove noise.

You're right that with repeated sampling you can learn more about the data set. If I understand it correctly, I think the solution to that in DP is that you have a limit to how many times you can make your repeated query before you have spent your privacy budget and are cut off. The idea is that for a given budget level you are limited to fewer queries than you would need to learn enough to de-anonymize the data set.

If your signal processing professor was already taking that into account then I would be curious to know how that attack would work.

In the US hospital systems are increasingly huge companies.

Reply on news.ycombinator.com