Hacker News Reader

Comment by gravelc | Hacker News Reader

Agree it's trivially easy to do, but as anyone you does a lot of qPCR knows, they're rife with false positives.

Any environmental contamination will show up as a positive, particularly using the non-quantatative end-point method mentioned above. There's simple controls for that, but still....

View on HN

> Agree it's trivially easy to do, but as anyone you does a lot of qPCR knows, they're rife with false positives.

Honestly we'd be better off having a widely available test that gives 50% false positives. Worst case, a bunch of people quarantine unnecessarily, but we at least catch all the infected. The alternative is what we have now: widespread community transmission with no end in sight.

With a 50% false positive rate, most of your positives are false positive. You might as well just tell everyone who asks for a test that they're infected.

Having 50% of all tests come back positive is way too high and makes the test basically useless. But having 50% or more of positive tests being false positives is probably the right tradeoff, though you probably want to re-test people who don't show symptoms.

From a public health perspective ... this could be a brilliant solution.

Until they all decide that they’re immune and don’t need to worry about it anymore.

Exactly. I think many people, upon learning they're positive by asymptomatic would (maybe) wait a week to make sure they don't feel ill, and then go out into the world, potentially spreading the virus with abandon because they're among the group that doesn't get ill from it.

You should account for ignorance in public health policy, but you shouldn't rely on it.

Having more false positives than true positives only works if the people taking the test either don't understand statistics or don't know the test accuracy. I for one would probably ignore a positive result if I thought my chances of true positive were 0.001 and my chances of a false positive were 0.5.

That's a great way to lose any remaining confidence the population might have on the establishment. It is also probably unethical so I doubt any doctor would go with it.

It's a perfectly valid concern, but I don't think it needs to turn out the way your fear.

How about we try this: when reporting "facts", how about we try noting that there is uncertainty involved? I know the average Joe and Jane public aren't intellectuals, and it is often how stupid people can be, but on the other hand I think it is also fair to say that we underestimate people.

I think we should at least consider trying this (noting uncertainty in reporting) - it's times like this where we need more ideas and more thinking, not less.

Until Fox News or the governor of West Virginia blows the illusion away.

Huh, I'm lost in this thread.

So, We lie to people and tell em they are infected. And then we are angry that FOX or the governor of WV are going to expose our scam?

What did I miss...

You could tell them the test came back positive. That's not really a lie.

50% is a bit of information. You can repeat the test - even if you consider that some contamination survives repeated tests, if you even get half a bit from subsequent tests you can easily get to eliminate quite a bit of population.

Yea but not nearly enough of the population for this to meaningfully change the response, and now you’ve made the testing results completely meaningless so that we can’t track the disease. Maybe it’s not intuitive to people but we are actually worse off with a test that’s this bad than just the extremely limited testing that’s being done now

I admit it's not intuitive. Tell me more?

I'm assuming there's high false positive rate. If the test has even moderate false negative, then yes it's worse than useless.

High false positive rate (I’m talking like more than 1%) means

(1) you are overwhelmingly unlikely to actually have the disease even if your test comes back positive but you don’t have obvious symptoms

(2) the pool of people testing positive is completely dominated by false positives. The fraction of people testing positive that will actually have the disease would be very small, so you lose the epidemiological benefit of the test (now you can’t actually track the disease well).

So — test is useless for people like you and me who don’t have symptoms or a strong prior probability of having covid BEFORE the test, because even a positive result is overwhelmingly more likely to be wrong, AND now the epidemiologists trying to actually track the disease are instead overwhelmed with false positives so the true signal disappears. It’s bad all around. You could imagine using this as a screening tool, but the false positive rate has to be small enough for that purpose as well (like, less than one percent).

There is a big system-level difference: with 50% false positive, half of the population goes back to living regular lives. False positives stay home, and go to a hospital only if they need a ventialtor (which very few will need since they are not actually sick)

There are 50% false positive, and there are some false negative. And some more not exactly false negative, but simply got sick after the test. And you tell them: we tested everyone, all negatives can just resume normal life. What could go wrong?

If you can't breath you go to the hospital regardless of it being COVID-19.

This is an interesting tradeoff to consider. Under many many circumstances I could see that rate of false positives being awful and intolerable.

In the context of a bona fide pandemic with a massive shortage of desperately needed testing, however, this would be a palatable improvement upon the status quo.

In the case of a generally mild disease, it's much less of a problem. Usually, the problem with false positives is unnessary, risky treatments and overwhelming the medical system. But in this case, you just need to know whether to self quarantine. You only need treatment if symptoms become severe.

Until the global economy unnecessarily collapses.

You also need to consider that when the false positives get over their flu or cold or whatever, they will assume they are immune to COVID.

Really? That test is meaningless. It screws up the numbers, and you might as well do a complete quarantine anyway, since you need to not just quarantine but isolate anyone who’s positive.

Let’s say there’s 100,000 people in the US with covid right now (like 10x the measured number)

FPR =0.5 TPR = 1

Disease rate = 1e5/3.3e8 = 3e-4

P(you have covid| test is positive) = 3e-4/(0.5(1-3e-4) + 3e-4) = 0.06%!!

This is less than useless. Now instead of tracking the disease you are completely overwhelmed with false positives and the result is exactly the same — everyone stay home.

hmm, how many false negatives is it giving?

A 50% false positive rate means that, since most people don't have the virus, about 50% of all the tests will come back positive. You just about might as well flip a coin instead.

This is a classic statistical fallacy. A test that is 50% accurate serves 0 purpose. You might as well consider coin flipping a valid test. If you tested America with this test you’d end up thinking half the country is infected. And within the ones that tested positive only half would actually be and within the negatives half would be and you would have missed them.

A test needs to be materially more accurate than the odds of having the disease to be worth anything.

Classic thought experiment: if 0.0001% of the population has a disease and a test is 99% effective and you test positive, what are the odds you have the disease? Answer: 1%.

Complimentary thought experiment: if an expensive preventive drug was available in limited supply (for 0.1% of the population only) and the earlier you took it the more preventive it was should you give it blindly to as many positives as you can? Probably not because 99% of that would be going to waste. And you would run out of drugs to cover the actual sick. Only 10% of the sick would end up actually getting the drugs.

Isn't the false positive rate the percentage of positives that should have been negative? E.g. if I test 10000 people and 100 of them test positive, but really only 50 of those are sick (and almost none of the other 9900 people are), then I have a false positive rate of 50% and a very useful test. Obviously, this is very different from flipping a coin. And flipping a coin could also have a very high false negative rate.

It's the rate of actual negatives that are perceived as positives.

I make that mistake myself all the time, What helps me avoid it is this: you want a number that is a property of the test method alone, independent of case distributions. A ratio between misidentified positives and identified positives (or true positives) would depend on sample distribution.

The metric you're talking about is called the "false discovery rate": https://en.wikipedia.org/wiki/False_discovery_rate

I also have a hard time keeping these all straight: https://en.wikipedia.org/wiki/Sensitivity_and_specificity#Ap...

50% accurate I'd agree is useless.

But the parent said 50% false positive, presumably with close to a 0% false negative would be VERY useful and save potentially millions of lives. We need enough tests yesterday or so to avoid a repeat of Italy and a 50% false positive rate (with a very low false negative rate) could help do that.

Even the properly-conducted version of this test has a fairly substantial false negative rate, somewhere around the 30% mark, and if it's done by students using samples that might not be taken correctly on a rigged-together testing setup that's going to get worse. Seriously, you might as well flip a coin.

A test with a 50% false positive rate, administered to 100 million people who aren't infected, would say 50 million people were infected.

False positive rate is a confusing term; it means the % of tests that should be negative that report a false positive.

I think the term for what you're probably thinking of (% of positive results that are false positives) is the false discovery rate.

That’s fair, if the assumption is that false negatives are very low or 0 (in which case it’s a super high accuracy test at low contamination rates and super useful).

And yeah obviously we desperately need a decent test like three months ago.

A test might have a high false positive rate, ie “low specificity”, but can still have “high sensitivity”.

That would mean that while you catch most cases of the virus, you’d also get a bunch of false positives. Flipping a coin as a hypothetical test would give false positives and negatives, or low specificity and low sensitivity.

Isn’t false negative specificity and false positive sensitivity?

https://en.wikipedia.org/wiki/Sensitivity_and_specificity

No, sensitivity is TPs / all positives (detected TP and FN). It’s the green half of the diagram in your link. The language is very easy to get tripped up by though. :)

I think I’m getting tripped by false positive, true positive.

A test with high sensitivity will have a low false positive rate.

A test with high specificity will have a low false negative.

So having a test with high sensitivity but low specificity will result in trust in the positives, but not trust in the negatives?

For me it’s easier to think of sensitivity as being “sensitive to the true positive” without saying much about false positive or negative.

A sensitive test will catch many positive cases. It may or may not have false positives though, eg a test that’s always gives the right answer vs a test that always returns positive no matter what.

A specific test will give you few positive results when the true answer is negative. You could use it to rule something out. One test might say “patient has A or B condition”, and a second test with high specificity may then rule out A or B, leaving B or A, respectively, as the probable condition.

I think you need to learn the difference between false positives and false negatives. A 50% false positive rate with no false negatives is absolutely better than 50% false negative and positive.

False negatives miss infected people and you fail to quarantine. False positives just mean you quarantine too many.

You’re not differentiating between sensitivity and specificity - in this case the test is highly sensitive but not specific. That means that it could be effectively hard to rule out cases while failing to provide strong evidence of actual infection in individual cases.

You're applying conventional knowledge to an unusual situation where it doesn't apply. The current test in my country is something like "was sitting within 4 rows of a known patient on a plane". That's obviously got a massive false positive rate but we do it anyway. The action you take when officially tested positive is the same as you take when learning you had contact with a patient - isolate yourself at home. There's no wasted drugs because there's no treatment for it! At worst, you might do an official test, but if those are in short supply, they're not going to be given to anyone who just walks in with a home-made result.

>A test needs to be materially more accurate than the odds of having the disease to be worth anything.

Shouldn't it just need to be better than 50/50?

>Classic thought experiment: if 0.0001% of the population has a disease and a test is 99% effective and you test positive, what are the odds you have the disease? Answer: 1%.

My odds still changed dramatically and the cost of a false positive is just me hanging out at home and not visiting my parents. I'm not getting a biopsy or taking expensive medicene.

Given one of the best pieces of advice about managing COVID-19 is to "Act Like You Already Have it" [1], a less accurate test with more false positives isn't necessarily a bad thing.

[1] https://www.huffingtonpost.com.au/entry/act-like-you-have-co...

Aren’t the primers supposed to be pretty selective for the RNA sequences being targeted? Otherwise any other random RNA in the sample (like from some other viral infection) would result in a false positive, right?

Once you have a successful PCR, you have billions of copies of your target sequence that are easy to aerosolize and/or transfer by contact onto your pipettes, gloves, and other lab surfaces. When a few of these molecules find their way into a new PCR you just set up, you get a false positive. There are protocols clinical labs employ to avoid this (e.g. never open the cycled PCRs in same room where you set up new PCRs, etc.) but they require strict adherence. (Or you can just give up all your sensitivity and only detect the cases with a huge viral load.)

Looking at the assay - it uses a dual labelled probe. That gives the specificity in addition to the PCR primers. Primers themselves presumably aren't selective enough.

> they're rife with false positives.

Is it possible that some of the positive cases of Corona are actually positive cases of Influenza?

No, it is not, see the numbers compared by the Italian National Biotechnology Association:

https://en.wikipedia.org/wiki/File:Is_COVID-19_like_a_flu%3F...

Each bar is one week.

Flu has totally different speed of increase and the amount of people needing treatment and dying.

We know what is the cause of the increase, the scenario repeats in many countries across the world, it's not one isolated setup.

I was curious about the possible mix up of flu and covid by the testing kit since they are similar, I am not denying that covid exists, just that if the tests are mixing the two up.

No, the covid testing kit was tested with more than a thousand probes and there wasn't a single false positive.

Source: Christian Drosten (he created the test) on the NDR Coronavirus Podcast

There is another source (in German) claiming the same, HN comment here: https://news.ycombinator.com/item?id=22627958 and source here: https://www.ndr.de/nachrichten/info/16-Coronavirus-Update-Wi...

> Regarding your question about if we might accidentally test for flu or other, already existing corona virii:

> This was a topic of discussion in yesterday's talk [1] with Dr. Drosten, a virologist who played an important part in the development of the currently used PCR test. He said that there were extensive studies done with hundreds of samples from both flu patients and patients infected with other corona virii and none returned a positive result. The only other positive results were from corona virii that are special to certain animals (bats, some cows IIRC), but none of those are present in humans. So the accuracy of our current PCR test for SARS-CoV-2 seems to be extremely high.

Edit: realize now that it's the same person (Dr. Drosten) so same source, but different format (podcast vs transcript)

> And theoretically, the test would give a positive result on some bat corona viruses, but they do not affect humans.

What if I have flu and that corona virus which do not affect humans. People will think I have COVID.

Edit:

The test developed was done against 75 people with various other diseases as listed below to find out the false positive rate. With the 75 samples (having listed diseases), the false positive rate was 0.

  Clinical samples with known viruses Number of

samples tested in all three assays HCoV-HKU1 2 HCoV-OC43 5 HCoV-NL63 5 HCoV-229E 5 MERS-CoV 5 Influenza A (H1N1/09) 6 Influenza A (H3N2) 5 Influenza A(H5N1) 1 Influenza B 3 Rhinovirus/Enterovirus 3 Respiratory syncytial virus (A/B) 6 Parainfluenza 1 virus 3 Parainfluenza 2 virus 3 Parainfluenza 3 virus 3 Parainfluenza A or -B virus 5 Human metapneumovirus 3 Adenovirus 3 Human Bocavirus 3 Legionella spp. 3 Mycoplasma spp. 3

   Total clinical samples 75

I hope we didn't any other major common virus, and also is 75 a big enough number to say that the false positive rate is definitely 0?

> What if I have flu and that corona virus which do not affect humans.

By definition, you can't have the virus which doesn't affect humans. The virus that causes Covid-19 is called SARS-CoV-2:

https://en.wikipedia.org/wiki/Severe_acute_respiratory_syndr...

2 means it's only a second ever recognized such. The first caused the original SARS in 2002-2003.

And that low chance of false positive can be easily derived (as in, just looking at the bars) from the Italian National Biotechnology Association graph, based on the known cases in Italy, which I've cited before. The numbers are orders of magnitude different, so the same can be concluded -- the can be no significant misrepresentation. There's the whole Italy and actual people dying or having to be connected to the machines to keep them alive -- a much bigger and more horrifying confirmation of low chance of false positives than just comparing thousand probes.

"mkagenius" sadly simply reiterates false claim dressed up as a restatement of the question after completely ignoring the answer to which he responds.

The viruses are not similar at all, the symptoms are to some degree.

SARS-CoV-2 is in the same family as SARSv1, MERS, a number of viruses that cause the common cold (229E, OC43, NL63), plus a number of bat viruses. But not influenza!

The test would cause a false positive for the most related ones, specifically SARSv1 and some bat viruses.

We really need a test for anti-bodies as well.

Antibody tests are also fairly unreliable in a number of circumstances as well.

Reply on news.ycombinator.com