Comment by davemel37 | Hacker News Reader

Comment by davemel37 | Hacker News Reader

Parent Full thread

Have them read Chapter 5 of this book. https://www.cia.gov/library/center-for-the-study-of-intellig...

"Using experts in a variety of fields as test subjects, experimental psychologists have examined the relationship between the amount of information available to the experts, the accuracy of judgments they make based on this information, and the experts’ confidence in the accuracy of these judgments."

"Key findings from this research are: • Once an experienced analyst has the minimum information necessary to make an informed judgment, obtaining additional information generally does not improve the accuracy of his or her estimates. Additional information does, however, lead the analyst to become more confident in the judgment, to the point of overconfidence. • Experienced analysts have an imperfect understanding of what information they actually use in making judgments. They are unaware of the extent to which their judgments are determined by a few dominant factors, rather than by the systematic integration of all available information. Analysts actually use much less of the available information than they think they do."

But, the main issue is "once an experienced analyst has the minimum information necessary to make an informed judgement" is followed by "they are unaware of the extent to which their judgements are determined by a few dominant factors."

Which implies they do not know which part of the data is most important. No?

disclaimer: didn't click link/read chapter

The authors ultimate solution is to use the scientific method to compare competing hypothesis...otherwise you'll be blinded by all sorts of biases. If you are running an experiment and using competing hypothesis...in theory, you can identify which metrics are best for that test... but if you had much more data, you'll probably end up using every data point past the first 5 as confirmation bias. (I'm no expert though and haven't read the book in about 3 years.)

Depends on the way of consuming that data. I agree it may be counter-productive to feed more data to humans, who are able to hold on only to 7-9 object in immediate memory at a time, and driven by subconscious reflexes hardwired into us by billions of evolution.

AI is not going to "think" like humans. It is not going to be better or worse, just "different". And all experience so far show that the more data the better are the decisions made by AI.

Cart, horse.

People are proposing smart whatever now, and handeaving its value proposition as -eventually- feeding into ML which will -eventually- have benefits to patients.

I don’t think there is disagreement that data + ML is useful, but the value for increasing data because of AI assistance isn’t meaningful in the absence of that AI. Hospitals and insurers have huge data sets available: build a halfway decent AI first, and then argue for the value-add of the data.

Those data sets are not public, hipaa and other issues make it difficult to build any sort of AI for this.

That is technically correct, the best kind of correct!

Except any department in any hospital or insurer that welcomes you to do research can make that de-identified data available to you with a simple contract.

How do you think college freshmen get in the door to work on research with physicians?

No, it’s not public, but it’s trivially accessible.

It's not trivially accessible. It's not impossible, but far from trivial. Startups like Tempus are doing it but I doubt it's trivial for just any startup to get that data

If I hadn’t worked in precisely that field and seen how easily we gave out data sharing agreements to partners, ...

But I did. No, I use the word “trivial” knowingly.

To partners, sure. What sort of partners ? If I start a company and show up at the door, would they just give me the datasets ? Would those data sets be in useful formats ? if you are seriously telling me those data sets are easy to get, let's talk because I know what to do with them.

There’s usually a spot in every insurance company that considers itself the “innovation center.” Basically, if you incur no costs to them beyond getting you the data, and give them a free trial run on your product, you’ll get your data-sharing agreement.

The data for insurers is in the form of Billings (which come with Diagnosis codes!), which follows a federally-standardized format (check out the Medicare website) and you can get a sample of 5% of Medicare subscribers’ data off the Medicare site without any hoops to jump through.

I want to say, “if you actually know what to do with it, PM me and we can talk details,” but the fact that you don’t already know about this sort of low-hanging fruit suggests you’re from outside the hc sector looking in.

I am from outside the hc sector, but I know how to build models on large data sets. So if you seriously know how to get the data, I seriously know what to do with it.

j s at self driving medicine dot com

Reach out, we can talk about what you’re thinking.

And yet, anybody who actually works in machine learning has repeatedly said "figure out what problem you're solving first, then determine what data you might need - don't just throw a classifier at a data garbage heap"

And the experience doesn't show more data is better, it shows that an excess of features leads to overfitting.

Sure, having more features gives a model more opportunities to overfit. But having more data points has the opposite effect, since they reflect the underlying distribution better and therefore provide a better estimate of actual model performance.

Reply on news.ycombinator.com