Hacker News Reader

Comment by waterside81 | Hacker News Reader

I work at a company that does applied machine learning and health care is the #1 source for prospective business. The hardest part isn't the analysis - it's getting good, valid data. Health care data is so sparsely collected, poorly structured (if at all) and the privacy issues surrounding gaining access are very strict (perhaps rightfully so).

The key, as IBM is doing, if working with a large HMO or health care network who hopefully have switched to a sensible EMR and have built up a good amount of historical data on patients.

I'd add one last key to getting this right isn't only the breadth of the data, but the depth. Knowing some superficial aspects of a person (age, weight, habits) is too naive. You need family history, you need psychosocial aspects (nightmares, trouble at work, marital problems etc.). If you can get that, then you're cooking.

View on HN

I'm also in this field and couldn't agree more. I hate to be a cynic, but the reason IBM is going down this road is almost purely because it allows them to capture a lot of poorly allocated health care resources (in essence, this is all about government rent seeking.)

Being a former IBM'er, I can tell you this, there is nothing altruistic in their motives. They invest billions into research with plans to reap the rewards. Health informatics, in Canada is brutal to say the least. See eHealth disaster. IBM employs some really intelligent people who intent to capitalize on this situation.

To clarify my remarks, I don't really have a problem with a profit motive... my argument is more that you could make money from (1) coming up with better treatments and selling them or (2) extracting poorly allocated resources from the government. We need fewer health care projects that are the latter.

I've worked on the problem of entering well modeled medical data for a decade and I have seen very slow improvement over that time. Not only is the overwhelming majority of medical data not in a proper data model, the data that is has most likely been subject to selective pressure (which will probably not be accounted for in most analyses).

Once you have enough well structured data, the next challenge will be reducing the dimensionality taking into account the meaning of the data. The fact that a patient is diabetic, e.g., can be inferred from a specific diagnosis, a set of medications and several lab tests.

It is a very promising field but I see non medical people entering it with a certain amount of cockiness.

I work in public health research and we collect similar data. We do it the hard way: surveys. Surveys dozens upon dozens of pages long. Because of the logistics involved and the time it requires to get the info from participants, the data is filthy. I can see the appeal of trying to get it direct from the insurance companies/government.

Agree, the more data points we get access to, the more correct analysis we'll produce at the end.

It's a pity knowing that there's a lot of information stored already in systems like 23andme, practice fusion, healthtap, etc. but I guess getting access to this data to aggregate it and analyze it would be... mission impossible.

Reply on news.ycombinator.com