Hacker News Reader

Comment by nevinera | Hacker News Reader

0.5% false positive rate is actually quite high. I don't have the numbers, but I'd be surprised if more than 1% of the people that would receive such a screening actually already have cancer, which would mean that it produces about as many false positives as false negatives.

Picture a population of a million people, all receiving the test. 1% of them have cancer (unknown to them) - half of those people get a negative screening result and half get a positive one (51% successful identification), and 0.5% of the full population get a negative screening result. Those numbers are roughly equal.

That doesn't make it useless by any means, but it's not nearly as 'specific' as it sounds on paper.

View on HN

I would very much like to know the confusion matrix for all these component tests: the false positive rate, false negative rate, i.e., the sensitivity and specificity of each.

For example, the Prostate Specific Antigen (PSA) test is famous for a high false positive rate, inviting many men to worry needlessly or get unwarranted biopsies. The current state of any test and its history are important to know for both patient and doctor, so both can have an informed dialog about the options after a positive result.

Also, each of these 50 tests are going to evolve over time, as will the accuracy of error rates for patient subpopulations. Race, age, ethnicity, co-morbidities and co-maladies will each shift the accuracy and precision of each test in ways that will make them much more useful if everyone stays well informed about all their merits and demerits.

There's an interesting discussion raging on elsewhere in the thread about the possibility for cultural adaptation that would enable us to cope with statements like "the cancer test was positive, but there's still a decent chance you don't have cancer".

There's also an interesting question, taking your roughly 1:1 true to false positive ratio as an example, of whether the marginal true positive does more good for the world than the marginal false positive.

Side-stepping those questions though, they probably won't give this test to literally every person in the population. How much does the picture improve when you give this only to people over 50? Or only smokers? You could still massively increase effective screening for cancer with a cheap and easy test if you combine it with enough population filtering to increase the true positive/false positive ratio to something like 70-90% rather than the 50%

That side-step was my actual assumption - I can't imagine that 1% of the _whole population_ currently has actionable cancer.

But yes, the higher you drive the prior, the less the false-positives cost. That's a fairly normal situation - it's why they have to evaluate the cost-value of further-investigation against so many different population groups to determine the actual optimal usage.

And yes, the question I was trying to prompt really is "what information would we need to have in order to know when this test is a net positive, from the patient's perspective" (and separately, "from the insurance company's perspective", since that's likely to have a very different answer).

I feel it's a net win anyway, if the procedure is not invasive and cheap, you could take one every so often (yearly?) and if something shows up you just double check with a better method or w/e. Better than nothing, which is the current alternative.

Right. If the objection is that it has a high false positive rate and doctors don’t understand Bayesian statistics enough to take that into account, then the solution isn’t to ban the test but instead require clinicians who interpret the test to learn Bayesian statistics better.

The problem is people are idiots. They get a false positive and their doctors tells them "well there are a lot of false positives with this test" but all people hear is "you have cancer." Then they get an invasive biopsy because the healthcare system is glad to take your money.

I appreciate you stating it frankly, I think a lot of people in this thread are arguing a variation of this point without saying it aloud.

A couple thoughts -

1) If this is the only/main objection, then the test is good news for those that are analytically capable, even if it's potentially bad for Joe Median.

2) While what you describe is how it probably happens in general in the USA, but I wonder if it plays out differently in, say, the UK / Canada under NHS/Medicare. In those systems you don't get to pick your treatments; your doctor will only prescribe a follow-up if it meets ROI standards. (Then the health system picks up the bill). Of course you can go private but there's a pretty big barrier there, and perhaps having your GP push back more strongly (as they would if you were not actually at high risk) that might prevent many people from inferring too much from a potentially-false positive result.

What if the followup procedure is invasive and expensive? It's possible that this procedure could be net negative in aggregate depending on the % of false positives and # of resulting needless followup procedures. Not to mention the emotional stress.

Exactly! This reminds of the recent case of the AMA increasing the suggested age of regular mammograms. More testing means earlier detection of breast cancer. But a false positive means an invasive biopsy. These are the two sides that need to be balanced. The AMA decided that since the base rate of cancer is so low in women under 50, the posterior probability that you have cancer given a positive mammogram for the under 50 age group is considerably low. Therefore they decided that the increased costs (monetary and emotional) do not outweigh the benefits.

From https://www.ama-assn.org/sites/ama-assn.org/files/corp/media...

"Screening mammography reduces mortality from breast cancer, including in women younger than age 50 years. However, screening mammography carries harms such as false positive results that can lead to additional imaging and invasive biopsy procedures, and overdiagnosis that could lead to treatment in patients who may not benefit from it. The USPSTF considered the balance of benefits and harms using a commissioned targeted systematic evidence review of randomized clinical trials and a decision analysis that compared the expected health outcomes of starting and ending mammography at different ages and using annual and biennial screening strategies; it concluded (in part) that routine screening should begin at age 50 years and continue biennially until age 74 years."

>But a false positive means an invasive biopsy.

Nah, there's plenty of other things you'd try before the biopsy.

Don't do it, then. Try homeopathy or whatever thing feels good for you.

So still get a second opinion just as you would have anyway.

It’s still getting caught

If it's cheap enough then get it performed 3 times.

Reply on news.ycombinator.com