Hacker News Reader

Comment by dbcooper | Hacker News Reader

The submission title has taken the liberty of heavily editorializing.

This is a submission to a minor journal, of a complicated material, attempting to do something that is clinically very difficult.

I'm not trying to dismiss this research, but please don't expect this to ever be clinically available.

View on HN

We've changed the title. Submitted title was "Urine Test Detects Prostate and Pancreatic Cancers with Near-Perfect Accuracy".

Submitters: "Please use the original title, unless it is misleading or linkbait; don't editorialize." https://news.ycombinator.com/newsguidelines.html

If the original title won't fit HN's 80 char limit, it can usually be shortened, as we did here.

From the paper and for contextualisation:

> In the same manner, the trained deep learning for pancreatic cancer urine dataset (Figs. S13a–c) clearly showed superior classification performance with a sensitivity of 98.6% and a specificity of 100% (99.3% accuracy, 0.9892 AUC, 59 epochs).

Seems strange to me that they put the epoch number there. How many people did they test?

My suspicion in reading the abstract and intro is that this is a translation issue. Inference informed by:

1) All the authors have South Korean institutional affiliations

2) "Detection of human biofluids such as blood, tears, saliva, sweat, and urine is important for clinical analysis of various physiological patterns" (First sentence, seems to be missing a word)

3) "differentiate patients from the normal group with high sensitivity and specificity" (abstract, patients and normal group is an odd way of phrasing this)

That being said...it had never occurred to me that nueral networks might be a useful way of interpreting spectroscopy data, that is a really cool insight

Neural nets operating on mel sprectograms (spectrograms shifted to bias frequencies that are of interest to humans) have remarkable ability for audio classification and synthesis. It stands to reason it would not be difficult to adapt methods towards nearly anything which can be mapped to an image in a similar way. I don’t think its particularly novel in concept and kind of surprised it isn’t significantly further along.

I think we are at the technology level to really make medicine significantly more affordable and available, at the cost of many high paying doctor’s jobs. Same can be said for lawyers. These are powerful groups that will not take being automated lightly.

Also from the paper for more context:

> The dataset was randomly divided into 70% training subset (1420 for prostate and 692 for pancreatic), 15% validation subset (304 and 148), and 15% test subset (304 and 148).

Please consider that the researchers reported a 99% detection rate of those particular cancers in normal people, using a urine test. One would be hard-pressed to come up with a less editorialized title, given HN's constraints.

I didn't read the paper, but in case you did: is it "near perfect accuracy", or is it 99%? The base rate of pancreatic cancer is very low, so a test with 99% specificity is going to overwhelmingly generate false positives.

The abstract claims that both sensitivity and specificity are high:

> The developed platform successfully classified the human prostate and pancreatic cancer urines in a label-free method supported by two types of deep learning networks, with high clinical sensitivity and specificity.

I don't have access to the full article so I can't see the numbers. It's not on the hub of science yet either.

It's not a sensitivity / specificity thing; it's a base rate thing [1].

Suppose that the percentage of people with prostate cancer at any one time is 1 in 1000 (i.e., then "base rate" is 1 in 1000). (Turns out this is actually on the same order of magnitude as real number [2].). And suppose this test has 99% sensitivity and 99% specificity.

And suppose you test 1,000,000 people.

Of those 1,000,000 people, 1,000 will actually have prostate cancer, and 999,000 will not.

Of those 1000 that actually have it, 990 will have a positive test (true positive), and 10 will have a negative test (false negative).

Of the 999,000 people who don't actually have it, 989,010 will have a negative test (true negative), and 9,990 will have a positive test (false positive).

So even with a test of 99% accuracy, if you get a positive result, your chances of actually having prostate cancer are still only 990 / 10980, or about 9%; 91% of the positives will be false positives.

And of course, the more rare the cancer, the worse it gets.

EDIT2: So, to follow on with GGP's point: "Near perfect accuracy" isn't very specific, but colloquially would imply that if you have a positive test, you have a high chance of actually having cancer. To get that number to 95% you'd need to have only 52 false positives, would require a specificity closer to 99.995%.

EDIT: Fixed some math

[1] https://en.wikipedia.org/wiki/Base_rate_fallacy [2] https://www.cancer.org/cancer/prostate-cancer/about/key-stat...

Right, the "lots of false positives" conclusion is still the correct analysis.

I just wanted to point out that both "P(Detect | Positive)" and "P(Not Detect | Negative)" are both high, since GP only mentioned one and not the other.

Since most people find probability pretty unintuitive, I figured I’d recommend an article that covers the math involved:

https://www.untrammeledmind.com/2018/01/cancer-screening-an-...

Indeed, Reverend Bayes’ dark side rears its ugly head right at the point that people claim that two nines of accuracy is good in the context of detecting a rare thing in a large sample set, without mentioning precision, recall, AUC, or other more useful metrics.

I think maybe the most interesting thing about it from my perspective is that deep learning is being integrated into this kind of technology. For all of the corporate immorality of Google and Facebook, the value to society of Tensorflow and Torch being free software cannot be overstated.

Is it a minor journal? They claim impact factor over 12. This is, of course, field related, but it's not nothing.

If I understand well the WP article about 'impact factor', it tells that in 2016 Nature was graded at 38 and Plos One at 3, so 'impact factor' might not mean too much.

And the criticism section of the WP article is quite long:

https://en.wikipedia.org/wiki/Impact_factor

The devil is in the details. Near perfect should be weighted against prevalence….

I agree that the HN title is editorialised, but can we talk about the actual paper? The paper’s title and abstract aren’t unreasonable, why is it that it will never be clinically viable?

Reply on news.ycombinator.com