MIT AI tool can predict breast cancer up to 5 years early | Hacker News Reader

MIT AI tool can predict breast cancer up to 5 years early

techcrunch.com

127 pointscodermobile31 comments

Take these results with a grain of salt. There's a large class imbalance in this dataset and ROC curves can be misleading in this case. The test set contains 269 positive examples and 8482 negative examples.

From [1]:

> Class imbalance can cause ROC curves to be poor visualiza- tions of classifier performance. For instance, if only 5 out of 100 individuals have the disease, then we would expect the five posi- tive cases to have scores close to the top of our list. If our classifier generates scores that rank these 5 cases as uniformly distributed in the top 15, the ROC graph will look good (Fig. 4a). However, if we had used a threshold such that the top 15 were predicted to be true, 10 of them would be FPs, which is not reflected in the ROC curve. This poor performance is reflected in the PR curve, however.

The authors seem to be aware of this in the supplement and also evaluate performance by a hazard ratio they define:

> We calculated the ratio of the observed cancer incidence in the top 10% of patients over the incidence in the middle 80% and referred to this metric as the top decile hazard ratio. We calculated the ratio of the observed cancer incidence in the bottom 10% of patients over the incidence in the middle 80% and referred to this metric as the bottom decile hazard ratio.

However, binning is a form of p-hacking [2]. And I'm still wondering why they don't just post the Precision-Recall curves.

[1] https://doi.org/10.1038/nmeth.3945

[2] https://doi.org/10.1080/09332480.2006.10722771

[Edit] to add link to [2]

It's one of the difficulties with attacking screening applications where the population TPR is very low. For screening mammo, it's less than 1 in 1000.

Without information about false positives, this is just basically saying they wrote an algorithm that sometimes points out cancer early. But if it is only correct 1% of the time, nobody is going to listen to it. It'd do even less than the current "You really need to check for cancer!" statements that we already have.

Edit: From the paper:

> A deep learning (DL) mammography-based model identified women at high risk for breast cancer and placed 31% of all patients with future breast cancer in the top risk decile compared with only 18% by the Tyrer-Cuzick model (version 8).

So better than before, but still only detects 31%. If I'm reading correctly, it's 95% correct? I guess that means 5% false positives? That wouldn't be bad.

> I guess that means 5% false positives? That wouldn't be bad.

Due to Bayes' Theorem, this can result in the majority of patients with a positive result being patients without cancer. Depending on what treatment is taken, and the psychological burden of erroneously thinking you have cancer, the negatives of the additional screening could outweigh the positives.

FWIW, it's not like the doctor is going to send you home with a diagnosis based on this screening. The psychological burden would persist only until further testing and analysis. And in any case, human radiologists have FPs and FNs, too. The question is how close and in what circumstances are we approaching current human radiologist performance.

No, the problem is work up rate. If you significantly increase the rate of biopsy (typically one of the possible next steps) eventually you are going to kill people, as biopsy is not a zero risk procedure.

You have to be really careful with FP rates in screening. Of course the answer to this could just be "don't biopsy due to only this result" but at some point that renders the approach pointless, if you can't find a cost effective way of mitigating the FPR.

95% on a 1 in 1000 event means that over 1,000,000 trials

950 true positives

~50,000 false positives

False positive breast cancer is pretty bad.

It seems they are simply assigning patients to the top risk decile (more accurately than before). Telling someone they are at a high risk is different than a flat-out diagnosis. Better efficacy of the predictive model (fewer false positives and fewer false negatives than before) is a good thing.

Cost of false positive is retest and observe, this seems like a very low cost. Also, stop consuming sugar.

Sorry, please explain to me if my interpretation is correct:

1. The cost of a false positive is low. 1a. The cost of a false positive is retesting (what is retesting? Re-mammogramming? Isn't healthcare in USA expensive?) 1b. Observe (so future re-mammogramming and hypervigilance).

2. Additional to the cost of 1a and 1b, cease consuming sugar. Which is a non-insignificant lifestyle change.

What I'm saying is... the statements "Rerun expensive tests and maintain hypervigilance seems like a low cost, in addition to major lifestyle changes that affect every level of one's socioeconomic place (eating out with friends/family, cost of food, food availability, habit breaking, possibly culture clashes, etc)." seems pretty much equivalent to absolute nonsense.

What about the anxiety and distress caused to the patient? How many might undergo an unnecessary mastectomy to be on the safe side?

I have a problem with this vision.

You can be told the risk factors for everybody, but when you are told the risk factors tailored to you it becomes dangerous? Should we tell everyone they're going to live forever so they live in ignorant bliss while healthy?

I can see being unprepared for having that information, but I don't think the solution is not having the information.

It's not a risk factor; it's a test outcome. You would be told that you've tested positive with the follow on tests and anxiety that causes.

There have been studies into exactly this: https://www.ncbi.nlm.nih.gov/pubmed/22859786

Specifically on breast cancer screening, the National Institute for Clinical Evidence in the UK has published management guidance that is quite interesting: https://cks.nice.org.uk/breast-screening

Can't read that last link outside the UK.

They wouldn't be wold they were positive for cancer. They'd be told that the computer predicts that they are at very high risk for breast cancer. That's knowledge that I think people should have, personally.

Given a positive result. P of having the disease are 2%. Given a false result, it’s about 0. Given 2 positive results, assuming independent results (I think that’s unlikely but let’s assume) then the probability of having it is 0.04.

I’m not disagreeing that there usefulness to tests, but there is danger too.

If by whatever mechanism your FPR significantly increases biopsy rate, it's not a low cost.

The false positive information is in the ROC curves such as Figure 2.

TLDR but 95% correct does not necessarily equate to 5% false positives: you can also have false negatives. The bottom line is the new algorithm increases the efficacy of the prediction.

But it said it only detects future cancer in 31% of patients, which means way more than 5% false negatives. I'm assuming that means that the 95% is only on the positives.

According to Craig Venter, early detection is what we need to eliminate cancer:

https://youtu.be/iUqgTYbkHP8?t=15m37s

The reason most people die from pancreatic cancer, for example, is because we almost always detect it in a late stage.

Early detection of breast cancer has had little noticeable impact on the mortality rates of breast cancer. Instead, it's been observed that the improved treatments for all forms of cancer have been the cause of a decrease in breast cancer mortality rates.

https://www.sciencedirect.com/science/article/pii/S221353831...

According to the CDC [0], the five-year net survival rate for localized pancreatic cancer is 37.4%. That's really low for an early-staged diagnosis. By comparison, early-staged colorectal cancer has an 89.9% net survival rate, despite the late-staged rate being 14.2%.

Granted, a small proportion of these cancers are caught an early stage. But I wouldn't say most people diagnosed would live five years longer if they'd caught it early. And that's ignoring lead-time bias [1].

[0] https://seer.cancer.gov/statfacts/html/pancreas.html

[1] https://en.m.wikipedia.org/wiki/Lead_time_bias

This depends on the type of cancer. In the case of a slow-growing cancer like prostate cancer, early detection can find cancers that would never be a problem (you'll die of something else first).

This has to be taken into account when deciding how much screening to do.

Not really. You’re probably still better off knowing you have the slow-growing benign prostrate cancer that you can monitor but not treat. Also, you might find you have the more aggressive form.

https://www.webmd.com/prostate-cancer/prostate-cancer-surviv...

Maybe the problem is that people don't do that? What do you think of this meta-analysis?

"Pooled data currently demonstrates no significant reduction in prostate cancer-specific and overall mortality. Harms associated with PSA-based screening and subsequent diagnostic evaluations are frequent, and moderate in severity. Overdiagnosis and overtreatment are common and are associated with treatment-related harms."

[...]

"Screening resulted in a range of harms that can be considered minor to major in severity and duration. Common minor harms from screening include bleeding, bruising and short-term anxiety. Common major harms include overdiagnosis and overtreatment, including infection, blood loss requiring transfusion, pneumonia, erectile dysfunction, and incontinence. Harms of screening included false-positive results for the PSA test and overdiagnosis (up to 50% in the ERSPC study). Adverse events associated with transrectal ultrasound (TRUS)-guided biopsies included infection, bleeding and pain."

https://www.cochrane.org/CD004720/PROSTATE_screening-for-pro...

many people also insist on getting antibiotics when they have a viral infection. We have a more measles cases now than a decade ago because of fake news.

You might be right that many people would be better off not knowing but I’d rather we took a step forwards than backwards.

Early diagnosis is much better than late diagnosis. Prevention with a healthy life style is even better.

Addressing model bias by adjusting which data the model has access to is a bad idea. Tweaking the data so that the model output looks equitable is going to make your model across the board. You should train your model on what you have and then add explicit biases to the classifier for different groups. That way you have the best model and are clear on your biases.

If this model is equally accurate for black and white women that means that either race is not a factor in predictability, that it is a factor but easily adaptable into a model, or race is a factor and they’ve reduced their ability to diagnose one group in the name of equity.

The linked article suggests accuracy gains are due to better risk models, that use more than age. I’m not sure if that means it’s tied into the image neural net. Would like to see false positive rate too.

Here's the paper in question.

https://pubs.rsna.org/doi/full/10.1148/radiol.2019182716

I wish they would call it something other than AI. Like "Diagnostics" or if there MUST be a buzzword there, then call it "Predictive Diagnostics".

Once upon a time, a necessary precondition to call something AI was that it should be something where there is at least the hope that it could one day generalize to pass the Turing test or something along those lines.

Medical diagnostics is one of the primary applications of pattern processing, and since it's pretty damned impressive as it is, it's a bit pointless to try and make it even more impressive by suggesting that you might one day enjoy a chat with your medical diagnostic tool over breakfast, exchanging views on how the Knicks' season is shaping up... (Which both the informed readers, and the people writing this, know pretty damned well is never going to happen, and was never intended to happen).

JFC this is great.

Reply on news.ycombinator.com