The Dangers of DNA Testing

Do you have a better alternative?

We need to foster a deeper understanding of how empiricism actually works, what its limitations are, and the difficult and often lengthy process through which varying degrees of certainty are obtained and adjusted over time.

I also think it is important to spread the value of grounded skepticism (including of one's own knowledge) and the power of freely admitting mistakes.

Obviously, none of this is easy. But it is worthwhile and I think small improvements will have a large impact through ripple effects over time.

We shouldn't throw the baby out with the bathwater, in regards to empiricism.

There are many people outside of that world who either downplay, or reflexively reject, any claim that comes from peer reviewed research, if they personally disagree with the outcome.

These folks won’t even consider academic peer reviewed articles as a legitimate source of information.

This viewpoint is disturbingly far reaching in my experience, in part due to poor reporting of science. I’ve had friends and family question if researchers of any discipline know much of anything because of frustration with dietary research reporting: “one month eggs are good for you, one month eggs are bad. Which one is it?!” They end up rejecting researchers and scientists of all types simply because of a paper here or there which got traction in mainstream media but was perhaps reported in a misleading fashion... and that does get to your point of how important quality journalism is.

I feel far more distressed by the attitude of “my ignorance is equal to their experience” than people pointing to a PDF of research and saying, “here’s the data,” which seems to be a piece of what you are criticizing. Am I understanding that correctly?

If people can read the study, they can question specific methodologies, rather than coming up with fun straw men such as, “well these researchers in this field need this to be the case, because otherwise they wouldn’t have a job.” I’ve heard that argument used on climate researchers and dietary researchers at a minimum, but those trends can extend to people’s thoughts where they start to think, “well those researchers are up to no good. I bet they have the cure for cancer and they just don’t want to release it.”

I’m all for ensuring people appeal to reason and not just take the hottest headline (be it in popular press or academic papers), but there is a problem with a large portion of the population that outright rejects many research papers altogether.

(I'm not from the US!) Is this the stuff like "arson detectives" working with completely unsubstantiated theories, often enough sending people to jail even if they were innocent?

I guess the quote works as a kind of content-based addressing, though.

Seems like the more I learn about the system the more of a scam it seems.

Speaking of which Serial season 3 just started which covers this kind of stuff, more in the small than a grand ‘Thing X is the problem we want solved that will make things much better’.

But the defendant themselves paid one of the experts, so...

Second, we have always known witness testimonies to be unreliable. That's why perjury is a crime! We wouldn't need it to be a crime if nobody was giving false testimonies. And that's assuming people don't make mistakes, which they do.

We still use testimonies.

A court case, after all, is all about probabilities. Very often we just don't know the truth 100%. It's just that compounding evidence, ideally, would convince that it is very unlikely that a particular scenario didn't happen.

E.g.: someone could have mistaken another person for you when they said they saw you hitting the old pawnbroker lady with an axe, BUT given that you were caught with an axe AND blood was dripping from it AND that blood matched the old lady's AND you wrote an article about the merits of butchering old ladies in the local newspaper - ALL those things could have been mere coincidences and mistakes, HOWEVER they establish that it's highly unlikely that you didn't commit the crime.

Same with DNA evidence. We just need to re-adjust our expectations of how foolproof it is for it to work effectively in a court system. As long as we know that the labs are not 100% reliable, we'll be fine.

And that's what the authors are trying to do here.

The next step would be punishments for quack/pseudo science being used as "expert" testimony, as well as actual experts being reckless in their work (as is the case with the DNA labs in the article) when that work is used to accuse someone of a crime.

The punishment doesn't need to be jail-time; even banning the person/lab from providing evidence again would be a good first step. But that's my view on how to solve this problem, and a subject of another discussion.

The particular issue here is the mixed samples. The basic approach of DNA fingerprinting is to look at a suite of variable markers which, taken together, form a unique set of alleles that can positively identify an individual. It's easy to see how a mixed sample can pose a problem: if you have two contributors and 20 loci analyzed, that's over a million permutations to consider, assuming all loci are different for those samples. Making an affirmative match in this situation is much more difficult! Not only must your statistics be much stronger (to offset the permutation complexity), but you must make far more assumptions about the sample than before. Namely, you must estimate the number of contributors to - and the relative proportions of - a sample via various heuristics to arrive at a statistical metric of match reliability.

It's kinda amazing that it can be done at all, but for smaller numbers of contributors, it's not that bad. The classic example is a sample that contains DNA from both the perpetrator and the victim; such 2-contributor situations seem to be well understood at this point.

The main issue for reliability is when there are a large number of contributors, and particularly when very sensitive assays are used. Increasing sensitivity means you have to rely more upon amplification, and the exponential nature of PCR makes it very easy for minor contributions to be out-competed.

Basically, not all DNA evidence is equal, and it's important to distinguish between more and less reliable methods. Unfortunately the vagaries of legal precedent are often have an outsize influence on what is accepted in court. The apparent reluctance by the NIST to report their findings is really what is most concerning here, not the tests per se.