Hacker News Reader

Comment by iandanforth | Hacker News Reader

Figures 3 and 5 from the paper (https://pubs.aip.org/avs/jvb/article/42/2/023202/3262988/Hig...) have overlap between all the groups. The same measured output could have come from any of the non-cancer, or known-cancer, participants. While the means of these groups look nicely separable, that overlap means there will be significant false positives or false negatives. So I'd label the studyfinds headline of this being 'accurate' to be false.

View on HN

Not at all. The model is quite accurate. In fact, with the distribution of samples that they have a model that predicts all cases as having cancer would also be very accurate. It would get 17/21 predictions right. The model lacks precision. I suspect that even with a fairly high cut off point the model would still produce a bevvy of false positive predictions due to that. It might still be useful as a screening step if they can increase the sensitivity further, but you would still rely upon further tests to get a true diagnosis.

It also only looks at HER2 and CA15-3 (aka MUC1) expression - what about breast cancers that don't express either of these?

I realize this is an early technology and I think it should continue to be explored, but I would anticipate that if compared head to head with screening mammograms, it would be inferior.

For patients with relapsed disease, this kind of technology would be neat to non-invasively re-assess biomarker status but as a screening tool, I find it lacking (and certainly a positive screening will require dedicated imaging and biopsy anyway).

Backwards. The distributions here imply it will be a high precision model with not great recall.

Sure we can be technically more surgical in our terminology, but GP is addressing the usage of 'accuracy' in the news headline. In that context, 'accuracy' is kinda a catchall term for both accuracy and precision.

It's a bit difficult to say, isn't it? The headline is using the term accuracy correctly, the reader might be ascribing the meaning you are to it, especially if they are non technical. As was the parent comment.

My goal in pointing out the difference was not to be snarky. It was to point out the very real statistical consequences. Any model can be accurate on a sufficiently biased dataset, but what matters once a screening test hits the real world are the precision (positive predictive value) and negative predictive value. These are the hurdles that the test will have to pass to see widespread adoption.

Exactly. This is the real test and so far we simply do not know the answer. It's a nice first step but the headline is simply not justified. But whether solar power, wind power or cancer it's 99.99% of the time (possibly more nines) far less impressive by the time all of the data is in. And that's fine, but headline writers seem to be stuck in a hype cycle.

Theyre not being surgical in the terminology. Accuracy and precision are the two things that matter for a diagnostic test. High accuracy, low precision = screener.

Reply on news.ycombinator.com