Hacker News Reader

Comment by motohagiography | Hacker News Reader

It's an important development. Forensic labs and companies are expert witnesses with black box processes and the incentive to protect the authority of their profession. They are as likely to lie as any other witness. Perhaps even moreso.

"Those arguing on behalf of the defense cited past problems with other genetic testing software such as STRmix and FST (Forensic Statistical Tool). Defense expert witnesses Mats Heimdahl and Jeanna Matthews, for example, said that STRmix had 13 coding errors that affected 60 criminal cases, errors not revealed until a source code review." "They also pointed out, as the appeals court ruling describes, how an FST source code review "uncovered that a 'secret function . . . was present in the software, tending to overestimate the likelihood of guilt.'"

An analogous situation with an alcohol and drug testing lab caused a scandal in Canada and called into question 16,000 child protective services cases: https://en.wikipedia.org/wiki/Motherisk , and then at the same hospital(!), a forensic pathologist was giving fake prosecution evidence, https://en.wikipedia.org/wiki/Charles_Smith_(pathologist)

Maybe there's just something about Toronto and compromised processes, but defense challenges to the integrity of automated systems looks like a growth field.

View on HN

> Forensic labs and companies are expert witnesses with black box processes and the incentive to protect the authority of their profession.

I imagine they're also incentivised to favour a result indicating guilt.

Why? The risk to their reputation & livelihood for a false positive should temper that.

While there have been cases of technicians guilty of systemic fudging of data, that is a tiny exception to the vast majority of honest technicians with self-respect.

Your sentimentality is not enough for a good justice system.

On the contrary, that's what the American justice system is entirely based upon. Fair judges, juries of peers and so on. Its the bedrock of justice.

Engineers have some notion that it can all be replaced with science. In so far as science can certify the reliability of tests, that is good. But in the end one has to trust the humans between those tests and the courtroom.

The whole point of this discussion is that this trust is being largely misplaced.

A lot of forensic science in itself is essentially phrenology (tooth prints, hair analysis - the non DNA kind, even fingerprinting is of exaggerated value). A lot of labs systematically turn out biased results with generally no expectation of risk, either personally to researchers and definitely not to higher ups.

Ultimately, forensic evidence should be seen as a signal, but not nearly as trustworthy as witness testimony (which in itself is not very trustworthy), despite what many believe.

And this is important for the general public to understand, the people who will participate in juries. The mystique that forensic experts have can make juries give extremely wrong decisions (even bad aquitalls on lack of forensic evidence).

Yes there is a need for controls and independent certification.

There is also a need to recognize that a lot of this science just doesn't work (to the extent that it is often presented). The right way to phrase a lot of the conclusions of these experts, at the state of the art, would be 'in my expert opinion, it looks like it might be their DNA/fingerprint/hair/teeth', not 'in my expert opinion it is [...]'. Because even the state of the art is often about at that level - probably closer to 75% accuracy than 99.99% like it's often treated. Especially on real world, partial, corrupted samples.

And to emphasize again, I'm talking about the state of the art without the biases being discussed. The biases only compound this problem significantly.

Why should that come to light if the process is opaque? Most defendants don't have the resources to challenge that.

And if it comes to light, acknowledge a one-time error and carry on.

I seems like there are no repercussions for sending an innocent person to prison. I don't know if that's true, but it definitely seems like it.

Depends on what you mean by repercussions. Since there is at least informal and sometimes fairly open scorekeeping happening at prosecutor's offices there's some hurt pride if your conviction is overturned later. Everything else either gets taken care of by the taxpayer or prosecutors' immunity for their official acts so on a personal level, no, there won't be any actual repercussions.

The prosecution might call in a different expert in the next trial if the current one is not 'good'.

Then they get hired by the defense.

The defence can rarely afford their services.

Prosecutors aren't necessarily blessed with bigger budgets in a lot of places, they simply have a separately-funded (and very well funded at that) investigative body that is the police or sheriff's office doing the really expensive part that the defense, especially a public defender's office, will only have a skeleton crew to do. The problem is that defense may not have the resources or knowledge to use inadequacies of something like this as a defense in the first place, and may not be able to effectively cross or direct an expert because of the lack of specific expertise and the reactive nature of the job.

It's not that the venn diagram of tech-literate and criminal-defense lawyers are entirely separate circles, but having been "that guy" in a public defender's office for even basic stuff like cell tower triangulation accuracy to finding proprietary surveillance video codecs to decode exculpatory evidence, you really need to start at square one while the prosecution have the whole police department's resources, expertise, and initiative at their disposal. You can afford the expert, it just won't do any good when you don't know what questions to ask that will actually be effective. And unsurprisingly those who do have experience in technical, specialized fields tend to get poached into the private sector or out of trial (really plea) practice all together, so the knowledge/bullshit gap will still exist and there's no real consistent way to bridge it.

Also the defense is many customers while the prosecution is bulk.

The one trial I sat in on, it was the public defender who hired the expert.

Go figure.

Because they likely use metrics which highlight how many criminals where caught due to their system.

On the other side a metric which says how many times no conclusive evidence was found with DNA testing sounds more like a metric of failure (but is not, if you don't have qualitative good enough evidence including "complete enough" DNA you should not make up results which simply don't exists, but then courts ruling without conclusive evidence isn't that rare and some would say even necessary to some degree. It's just that many believe DNA tests are somehow unfailable perfects ways to prove the presence of an specific person at a specific place, but they are often not).

The hands-off reputation-based approach isn't enough to regulate food and drugs, or aviation. I don't see why for-profit forensics companies should be trusted implicitly.

Also related: we know that funding bias [0] is a real problem in science, despite that scientists' reputations should (ideally) motivate them to resist such biases.

[0] https://en.wikipedia.org/wiki/Funding_bias

No, the question is what is the incentive for lying and claiming guilty? Do prosecutors choose different labs?

If you knew someone was guilty, because of a preponderance of other evidence you have the utmost faith in, such as a professional law enforcement officers’ eyewitness account, and the DNA testing vendor you were evaluating said “not guilty”, what exactly would you expect to happen?

That the eyewitness account would be discarded, they make for terrible evidence anyway.

I think it is REALLY hard to overstate how dangerously naive a view of the american justice system that is

I don't understand what you mean. I would expect them to make the obviously correct decision, though I'm fully aware that they very often don't.

It depends on the definition of 'expect'. You should expect it of them, as in consider it unethical if they don't. You shouldn't expect them to live up to this standard, as in, if you were to bet on it, you'd likely lose.

That seems plausible.

I would argue that if prosecutors and/or cops get to pick the tool, false positives would result in more sales.

Just as false negatives would result in more sales if the defendant gets to make the pick.

false positives would result in more sales.

That seems highly unlikely. Prosecutors, like all attorneys, talk. It wouldn’t take too many getting burned by a false positive before word would get out (regardless of any nda).

For example, once an expert gets Dauberted, you’ve got to think long and hard about being the next guy to use them.

But how would one actually know if a result is a false positive or not?

Sure, in extreme cases it might be obvious, but if you just know that one test gives more matches than another, not knowing which one is the one giving false results?

But genetic evidence is often enough to convict by itself. A false positive is far more likely to land a conviction than an embarrassment to the prosecutor. I mean we live in a world where DNA evidence will easily override solid alibis.

A product that doesn't work wouldn't be considered.

Paying investigators per conviction is a sure way to create incentive.

Exactly. "Product X caught 25% more bad guys than Product Y" this stuff just writes itself.

Policing is driven by numbers. There was a whole season of The Wire about this.

Of course prosecutors will favour expert witnesses that will help them get convictions. Why isn’t that obvious?

There are countries where the job of the prosecutor is to uncover the truth, not to convict someone at all costs:

https://en.wikipedia.org/wiki/Inquisitorial_system

I know. I live in such country. But even here it is better for the prosecutor’s career with many convictions.

I doubt there's any system in the world where it's officially the job of a prosecutor to "convict someone at all costs". It certainly isn't in the US. Nevertheless....

Most of the very real dynamics that affect all of our lives at a policy-making / systemic level are never official. If you think the official stuff is a reflection of reality ... I don't know how to finish this sentence. Safe from it being "official", it is very much the job of a prosecutor in the US to convict someone at all costs. This is common knowledge.

The Soviet Union is a perfect example of this. They had set up a proper republic. It had a constitution, elections, there were government posts etc. The problem was that the Party came first. The official that held the government post might've had the legal power, but in reality they were rubber stamps for the Party.

Conviction rates are surprisingly high in a lot of the first world.

I’m not sure what conclusion we should draw from that isolated fact. It can mean that people are convicted on too weak evidence, and it can mean that people are only put on trial when the evidence already is very strong.

I think we can look at the plea bargain rates and assume that most people do not have the resources to stand trial, and thus plea out whether or not the charges against them are valid.

Ok, maybe I’m naive, but why is that obvious?

Not arguing just trying to understand.

My rationale is: prosecutors have to weigh an “easy” conviction vs the possibility that labs that always guarantee results may be biased. In the event that they get caught up with a biased lab, wouldn’t it look bad for their reputation? As in, they can’t even perform due diligence on their sources, how can you trust them in a promoted role?

Or am I just naive on how the legal system works?

I think you're underestimating the usefulness of scapegoats. If biased results exist, a lot of people look more effective. If such a bias is discovered, only a subset of those people will be blamed.

> In the event that they get caught up with a biased lab, wouldn’t it look bad for their reputation? As in, they can’t even perform due diligence on their sources, how can you trust them in a promoted role?

I am not a lawyer but as I understand the U.S. criminal justice system (which is adversarial between defense and prosecution with the judge officiating and the jury picking the winner) prosecutors are not experts in particular technical fields which is why courts allow expert witnesses. The defense is responsible for calling their own experts who can dispute other biased witnesses. If no one successfully disputes an expert witness over time the prosecutor's trust in them is very likely to grow.

A prosecutor will always claim they did due diligence by selecting a reputable expert witness up until it's proven that the witness was not in fact reliable, but claim (likely in good faith) that they had no way of knowing the expert witness was biased.

A better solution might be to mandate that expert witnesses get independently tested but again that's something a defense council could facilitate and bring into evidence. If no defense team finds it valuable to do this kind of work then it's unlikely a prosecutor will see it as useful either. Lack of challenges implies that the evidence is more or less indisputable from a legal perspective.

I wonder how much of this has to do with funding differences. Public defenders are famously underfunded but prosecutors have essentially unlimited resources. As a result they are on a fast-track to political fortune, especially if they "do their job well" and "win" (get a lot of convictions).

How can the truth-finding benefits of the adversarial system be preserved while leveling the playing field by equalizing resources?

My naive thought is that both parties pay into a common fund that is split equally between the prosecution and defense. But that seems like it has it's own set of pitfalls. Are there other models we can look to for ideas?

So there's an underlying problem here, which is that at our current rate of indictments, our court systems cannot support everyone going to trial. One of the reasons people in charge of funding want to starve the public defenders is that if they're actually good, people will be less willing to just plea out to a lower charge. So the growth will be non-linear as their success rate increases due to the funding, which they now need more of to service the extra cases they get from being good.

One idea is that whoever introduces an expert witness must also pay for the other side's expert witness, up to the cost of your expert. The defense doesn't need to prove anything due to the presumption of innocence, so the defense gets free counter-experts and only have to pay for the counter-expert if they want to bring up an expert about something the prosecution doesn't want to talk about.

Huh, that’s an interesting idea but it seems to only address one form of inequality. Clearly the incentives are perverse. If there are more cases brought than the system can handle then this just sounds like a band aid at best.

The real issue is why are there so many cases?

>I am not a lawyer but as I understand the U.S. criminal justice system (which is adversarial between defense and prosecution with the judge officiating and the jury picking the winner)

That process happens very rarely. Federal criminal cases are resolved via plea bargain in ~97% of cases and state criminal cases are resolved via plea bargain in ~94% of cases[0].

This is a travesty of justice, especially since most defendants are severely over-charged and often face long prison sentences if they actually insist on (and can afford) a trial.

Those practices, along with cash bail, force even innocent people to plead guilty to avoid having their lives destroyed by bankruptcy, loss of employment, homes and even custody of their children.

And once they have a criminal conviction, they are stigmatized for life and are shut out of many jobs.

In most of these cases, the forensic evidence (if any) is never presented. For a discussion of this, as well as American forensic practices, see Blood, Powder and Residue[1], by Beth Bechky (Author discussion can be found here[2]).

While there are no required standards for forensics labs in the US (with some exceptions[4]), there are accreditation programs (example[3]), and not all forensic laboratories are for-profit companies.

I'm not defending the quality or independence of any particular forensic lab, but it's definitely more diverse than just a bunch of corrupt, rapacious scum sucking at the teat of prosecutors' budgets.

That said, most criminal defendants are at a significant disadvantage when it comes to performing/challening forensic research, as many state and federal labs provide such services for prosecutors, while defendants need to pay, often through the nose for them.

It's just another way the US "justice" system is stacked against criminal defendants.

[0] https://www.nbcnews.com/think/opinion/prisons-are-packed-bec...

[1] https://press.princeton.edu/books/hardcover/9780691183589/bl...

[2] https://www.c-span.org/video/?508130-1/blood-powder-residue

[3] https://anab.ansi.org/en/forensic-accreditation

[4] https://www.justice.gov/opa/pr/justice-department-announces-...

Edit: Added notation for reference [1]

I'll take a different approach than the other folks responding to you -

There are plenty of profit motives involved that have little to do with securing future contracts with police departments.

At the most basic - Source code reviews are expensive. They can often throw thousands of false flags that developers have to individually track down and verify are not problems.

Then there's the reputation hit of admitting a mistake - If you've had a bug for the last 5 years that makes you less reliable, that alone is likely to impact future contracts. So given the option of revealing this, admitting the mistake, and tackling the cases it impacted... OR... simply sweeping it under the rug and fixing it internally - some companies will take option number 2 (possibly most companies).

Prosecutors don't seek justice, they seek a conviction.

Sometimes the expert witness is in the department's own lab. It seems a tautology that they will favor conviction. Every officer bringing in evidence is begging "Give me something I can use to get this scumbag!" The pressure to fudge is enormous.

A fairly simple step is to certify equipment independently, and to do blind testing (one sample is provided from evidence; another by an independent unrelated source). Of course this costs more, but anything else is effectively an experiment without a control.

Yes, yes they do. Prosecutors often campaign on being “tough on crime”, and even judges (where elected) are known to sentence more harshly during election season.

The ability for an expert witness to deliver more convictions, even if by lying, is a feature for prosecutors, not a bug.

You're a prosecutor. Your goal is to get convictions. You are objectively graded largely on your ability to secure convictions.

Every incentive you have leads you to preferring convictions.

You have a choice between two labs to hire for your field test. Lab A and Lab B.

Lab A gives you conclusive evidence leading to a conviction 95% of the time. Lab B gives you conclusive evidence leading to a conviction 65% of the time. Price and speed are roughly comparable between the labs. Which lab do you select?

---

You are now a lab director. You are under pressure to improve the sales of the lab's forensic services. You have pretty much maxed out making changes to improve price and speed, and have been matched by other labs. You understand very well the decision making process prosecutors will use to hire your lab.

---

The incentives clearly lead prosecutors to pursuing labs that deliver better results. Labs know this, and so are under market pressure to delivery convictions for prosecutors.

Perhaps in a just society. Most places, the state and it's agents can do no wrong, there's almost no incentive to do the right thing.

A lot of DA and other prosecutors are measured by the number of convictions they get vs the number of cases they take to court.

Their goals are always to get plea bargain because that is automatic guilt or if they take it to court, they'd better win. That's a lot of state resources just to lose.

Overtime, you'll get people willing to bend the rules, turn a blind eye to 'weak evidence' or whatever.

You're only as good as your measurement/incentivisation.

Deal: Parole for a year. If not, full resources of the government to convict you, jail time in the decades. Evan a failure to convict can ruin you for life.

Even innocent people with a good alibi are going to take the deal.

Prosecutors want a 100% conviction rate. They are the ones who hire these companies. Generally speaking companies give customers what they are looking for.

When a DNA lab issues a false positive, its CEO doesn't go to prison, nor do they get hit with a charge-back for all the tests they've done in the past year.

Their reputation doesn't matter as much as you think. Police procurement isn't based on the reputation of the vendor. It's based on whether or not the vendor delivers the results they want.

In fact, government procurement in general, isn't based on the reputation of the vendor. This is by design.

This is a long tail regulatory force, but unfortunately, does nothing to stop bad actors from jumping in and making a quick profit for as long as they can until it goes noticed.

Society also no longer has tolerance for processes that are not fully auditable and transparent, especially when it comes to our already marred justice system.

Livelihood is frequently the reason why biases creep in. Independent labs tend to do better.

Here's a look at two different Texas crime labs. Compare and contrast, and see if you can spot the differences.

https://gritsforbreakfast.blogspot.com/2021/02/beyond-aid-to...

It starts with one pledging to:

   aid law enforcement in the detection, suppression, and prosecution of crime

whereas the other seeks to

    provide medical examiner and crime laboratory services of the highest quality in an unbiased manner with uncompromised integrity.

and goes from there.

Maybe for some, but look what happened in Massachusetts. https://en.wikipedia.org/wiki/Annie_Dookhan

The risk to their business of negatives whether or not they are false negatives, is likely higher. The resource asymmetry between a tax payer funded prosecution and a defendant funded defense is enormous.

> incentivised to favour a result indicating guilt

Exhibit A: Annie Dookhan of the Massachusetts State Crime Lab who is suspected of falsifying thousands of drug tests

[0]https://www.cbsnews.com/news/massachusetts-lab-tech-arrested...

Annie Dookhan did about 3 years of jail, which is risible, considering that her cooking of evidence affected > 30000 cases. No visible consequences for the supervisor, which is scandalous, considering that the had been carrying on for at least 4 years.

They fill a similar role to prosecutors that field drug tests serve to cops[1]: they aren't meant to be accurate, they exist to give the cops legitimacy as they proceed to do whatever it is they wanted to do to you. In the cops' case, they need legitimacy to search and arrest you, and in the prosecutors' case, they need legitimacy to charge and convict you.

If field drug tests were actually accurate, they wouldn't be bought and used. Similarly, if an expert witness fails to give prosecutors the results and testimony that they want, they wouldn't be hired again.

[1] "How a $2 Roadside Drug Test Sends Innocent People to Jail": https://www.nytimes.com/2016/07/10/magazine/how-a-2-roadside...

> They fill a similar role to prosecutors that field drug tests serve to cops[1]: they aren't meant to be accurate, they exist to give the the cops legitimacy as they proceed to do whatever it is they wanted to do to you. In the cops' case, they need legitimacy to search and arrest you, and in the prosecutors' case, they need legitimacy to charge and convict you.

In case any of you are interested, or completely flabbergasted as I was, by the idea that law enforcement's purpose and raison d'etre could ever become as distorted and contorted as this comment mentions, you should definitely read more on Walter "Johnny D." Macmillan [1] or watch the movie based on his story: Just Mercy[2].

Absolutely mind-blowing stuff.

[1] https://en.wikipedia.org/wiki/Just_Mercy

[2] https://en.wikipedia.org/wiki/Walter_McMillian

Wow, think about what it takes to be an elected official (Sheriff) and not only perjure, lie, and coerce witnesses, but be outed for all of those crimes, and then continue to run for election and continue your tenure for decades. How does someone live with themselves knowing they purposely put an innocent man on death row, and then CONTINUE YOUR CAREER, probably thinking yourself as successful?

The guilt I feel when I introduce a software bug that effects my corporate customers can sometimes burn my motivation for WEEKS. Who are these people?!

This absolutely depends on the agency/situation.

My agency used to rate our agents based on the amount of cases that led to convictions. Years back, we recognized that disproving an allegation was equally worthwhile, and settings promotions/evaluations based on "proving or disproving" the allegation was much more objectively reasonable than focusing strictly on convictions. There is still a reference to case completion speed, value of recovered money/property, and conviction results (providing a thorough enough investigation that the suspect is convicted for X years, for example), but disproving an allegation or proving that a different person committed the crime is far more appreciable to the agency.

We're fortunate in that we are a federal agency, instead of local/state level, where they can be significantly more busy with countless lesser offenses, while still running some extremely high-profile stuff, but I think it would be a huge boon to law enforcement if states mandated an approach focused on this objective metric that isn't strictly on "getting him".

Imagine what an innocent man goes through.

If I was innocent, I'd be 100% certain that the system was screwed up, and if DNA evidence "proved" it, I'd want to look at the code too.

I remember reading about the husband who was falsely accused of murdering his wife, and it was very very sad.

https://www.texasmonthly.com/politics/the-innocent-man-part-...

DNA testing should be double blind. I have no idea if it is.

Sample contamination can be arranged before sent to the lab though.

i think we can all agree that black boxes are not desirable.

But let's not jump the gun here, and assume ill-intent.... to explain something that can be attributed to negligence, or simple human error.

Why shouldn't we assume ill-intent? The precedent and the incentives are there.

I agree that we shouldn't assume ill intent, but we should recognize it is a very real possibility. Ill intent comes in when you realize the simple human error. Alone, in your office, poring over code or reviewing past files. Then you weigh the impact on your career and reputation of honestly fessing up, and put it off for a while, then a while longer.

When your negligence results in people wrongfully spending decades in prison I would argue that it ought to be criminal negligence. If your software does not produce results that are accurate, precise, and have confidence intervals included, how OK are you with people going to prison based on them?

Yeah, there need to be consequences for this. The alternative is empowering people and corporations to destroy lives with impunity in pursuit of their own goals, as long as they can later claim it wasn’t intentional.

What really pisses me off is how we fail again and again to ask the question of whether these entities should have been doing what they were doing in the first place, if it even can be done safely. Facebook, Equifax, police surveillance and misconduct, totally unaccountable “forensics” techniques—in all these cases where incalculable damage is done in totally predictable ways, we only address it reactively (if we address it at all) and completely fail to fix or even really consider the root cause(s). It’s like we have this huge blind spot where we take as a given that established entities doing thing(s) just have a right to keep doing them, regardless of any other factors.

In this case, forensics “experts” (essentially witch doctors) and fly-by-night black-box “labs” with unproven methodologies an undergrad intern might have developed in a week or two are institutionalized in our courts. This has happened because there is zero oversight and zero accountability for their claims, and lives are destroyed as a result.

It is insane not to pursue criminal charges for these indiscriminate arsonists of justice—if they can’t do what they claim to do fairly and accurately, they should not do it. They don’t have a right to “try” just because it will make them money, regardless of the consequences.

A lot of people respond to this as if the fact that it would be nice if forensics worked justifies treating them as if they do. 'What do you want them to do, stop looking for fingerprints and DNA evidence?' Yes! Just like I don't want them to use 'lie detectors' or hire psychics.

Exactly! It’s pure wishful thinking, and in a lot of cases those wishes are (at best) frighteningly amoral in context.

It’s like a hack to get around the whole “reasonable doubt” thing: just sequester all that pesky doubt in a black box so that people can’t see it, and if anyone asks just play dumb.

The quality standards need to be set by the courts or by leglislation.

No software is perfect. It's unreasonable for any bug to be considered criminal negligence, or else the entire software development profession would be in prison.

Instead, software that can result in life or death or prison scenarios needs to have tested, documented, verified quality controls, just like we do for vehicles.

But that means courts or the legislature have to decide what those standards are.

>I would argue that it ought to be criminal negligence

I would like a thorough analysis of everything you've ever learned that led you to this conclusion, and if we find one mistake in your lifetime of learning then this is also criminal negligence on your part.

> a forensic pathologist was giving fake prosecution evidence

The doctor handled thousands of cases in his career, and a well-funded inquiry found issues with less than 1% of them.

Sure we'd like that to be 0%, and society should spend time thinking of how much they rely on complicated processes of reasoning, but that's a really good accuracy.

You say that as if these were mistakes or unfortunate culminations of circumstance. The man committed fraud and sent innocent, grieving parents to prison by proxy. I don't think it's unreasonable to expect that to be 0.00% and punish those harshly who exceed that metric.

If I'm a juror, and the evidence presented happens to have a false positive rate of 1%, that certainly meets my understanding of "a reasonable doubt."

I would need a very very big pile of very clear cut evidence to find someone guilty of a crime where the likely outcome is a substantial prison time.

It's all tradeoffs, but "his blood was found on the murder weapon" wouldn't be sufficient for me - some childhood enemy could easily have planted a bunch of forensic evidence next to a crime scene.

Unfortunately, the vast majority of crimes don't produce evidence that rises to the level you call for.

After all, if my next door neighbour barged into my house and beat me up, the only evidence would be my visible injuries and my statement that it was my neighbour.

I'm not sure if I'd prefer a society where he would be convicted, or where he wouldn't.

Maybe your neighbour has a nicer car than you which makes you feel insecure so you decide to beat yourself up and go to the police saying your neighbour did it.

Considering that possibility, I don't want to put your neighbour in prison.

And presumably, if having been denied justice I pursue revenge instead, I can barge into his home and beat him up, as he did to me, and avoid punishment for the same reason?

While I admire your devotion to Blackstone's Ratio, this doesn't seem like a recipe for stability and rule of law to me.

A murderer kills less than 1% of all the people they meet. Does that mean we should just accept their behavior?

A murderer who kills 1% of the people he meets has certainly committed a crime. A person who has a 99% chance of being a murderer has not certainly committed a crime.

That is depending on your threshold of certainty. 1% is not that high considering that, according to the OJJDP, 5 milion people were arrested for serious charges in 2019 so with a 1% false positive rate that would be 100,000 people falsely imprisoned every year.

It does not have to be and really can't be 0% but 1% is unreasonably high in my opinion. If it can't be helped then it can't be helped but that isn't necessarily the case with these devices.

The doctor did not have the _potential_ to fake evidence. He _did_ fake evidence.

Is it bad that I'd guess you would be excluded from a jury if you admitted this upfront?

Additionally it could be made a requirement that, like in professional sports, two or three blood drafts are taken - one sample is analyzed and one is either kept until the case is closed or analyzed at a different lab if the evidence is contested.

This method doesn't protect against evidence tampering. Someone can just take a drop of that blood and smear it on the victims clothes. Now it doesn't matter how many labs analyze it, it's always going to come back a match...

The wikipedia page is heartbreaking. Griefing parents sent to jail, their other children taken away and placed in foster care... So many lifes made miserable by one person...

If the error goes against innocent people, then it must be unacceptable. If the error sides with the “would-be” criminal, that is more acceptable as we, as a society, have decided that letting a few bad folks go is preferred to imprisoning innocents.

> as we, as a society, have decided that letting a few bad folks go is preferred to imprisoning innocents.

We, as a society, like to say this because it makes us feel good.

But a critical analysis of our actions reveals that we don't believe it for a second. We are happy to write overbroad laws that allow most behaviors to be criminalized, over police marginalized communities, and place those who have been convicted by this flawed system in deeply dysfunctional prisons that may well violate their human rights.

From the Wikipedia page

> In June 2005, the Chief Coroner of Ontario ordered a review of 44 autopsies carried out by Smith. Thirteen of these cases had resulted in criminal charges and convictions. The report was released in April 2007, indicating that there were substantial problems with 20 of the autopsies.

That sounds more like 45% to me.

Also this bit from the article about a baby that allegedly had a skull fracture:

> Later exhumation of the child and examination of the skull have shown that there was no skull fracture. It is thought Dr. Smith confused the normal gap between the baby's skull plates for an injury.

Holy shit that is bad. Even as a lay person I know about these gaps.

> there are gaps between skull plates.

Between Baby skull plates, not adult ones ;-)

They grow together at some point.

I edited my post to make it clear I don't think skulls have gaps in general.

> Holy shit that is bad. Even as a lay person I know that there are gaps between skull plates.

Sure, but can you tell them apart from a fracture caused by injury? Presumably someone knowledgeable in medicine should be able to, but maybe it is one of those things that isn't as obvious as we, the uneducated outsider, might think.

That's an extremely flawed argument. If you can't explain it to a layman then you don't understand it. If you can't tell whether it's a gap or a fracture -- you can't get a conviction.

"It just looks like it" is not a valid reason.

Interestingly there's an area of psychology called Naturalistic Decision Making which studies how experts make decisions that they can't explain. (Example: a firefighter may be able to pinpoint where a fire is before they enter a house and see it.)

I know, but that is reason for suspicion, not evidence for a conviction.

It is possible that they could have well supported their assessment with evidence and still been wrong.

The end result of that tiny little error rate was tens of millions in settlements and legal costs, families permanently separated by government mandate, and multiple people spending more than a decade in prison.

I'm not so sure we should be allowing challenges to expert witness, with no evidence. That sounds like a rich source of abuse of the system, like a big company filing thousands of briefs to delay a case.

It's important for experts to use validated, tested equipment. That's clear. And by the sound of it that is happening (stories of how code was tested and found wanting confirm that; stories of double-blind testing).

Maybe what is needed is, an 'underwriters lab' certification for such devices. To forstal the inevitable gaming of the 'right to challenge'.

In most criminal complaints the defendant has far fewer resources than then the prosecution. The prosecution has a conflict of interest to get a conviction. Absolutely everything about their process needs to be above reproach, including their experts. A challenge to that process, approved by a judge, seems like a small consolation in a biased system.

When I was a juror on a murder trial the one thing that was most clear was the disparity in resources.

On one side were 6 impeccably dressed men in what I presumed were multi-thousand dollar suits. On the other side was single overwhelmed, overworked, and under-dressed defender who was steam-rolled at every sentence she spoke.

I ended up in traffic court last year for a parking ticket that I can only think must have been stolen from my car during a festival. After spending just a few hours waiting my turn, it was pretty clear who exactly the police target for traffic violations. But this is a tangent.

Grossly oversimplifying it, but it is what my lawyer uncles say, and is consistent with what I saw as a juror on a civil trial involving many topics I'm personally an expert in, expert witnesses are paid to lie on the stand. At the very least they are paid very very well (the amount they were paid was emphasized at the trial I was at) to state only the information consistent with what the person paying them wanted to them to say, regardless of whether they know information that would be more helpful to the other side.

Of course this is the case, but rather than lying, I think it is only a consequence of selection bias. You don't see the experts rejected by the prosecution (or defence) for opinions that aren't helpful, because of course they won't be invited to testify.

> At the very least they are paid very very well... to state only the information consistent with what the person paying them wanted to them to say

Anecdotally, if you replace 'paid' with 'incentivized' my understanding is the same.

And here's a totally unrelated Wikipedia link: https://en.wikipedia.org/wiki/Michael_Baden

They answer any question put to them. The defense has to ask the right questions of course.

I know expert witnesses. They are of impeccable character, and will seek to educate as well as they are allowed.

You know some expert witnesses. I doubt you have a good acquaintance of a significant percentage of all expert witnesses.

The experience I had, which was pretty cool (although the case overall was very very depressing) in that we the jury were allowed to anonymously submit questions for the judge to ask the witnesses as well."They answer any question put to them." definitely did not apply to all of them. There were many, painful to watch, who at best gave half answers. I'd say the expert witness in the movie "my cousin Vinny" who testifies that the tire marks between the two cars are identical and that it is possible for both cars to have made the tire marks, while knowing that it was literally impossible for the defendants car to make the marks on the road, is very very close to what I saw experts do. In my case it was dumber, though. We had experts talking about how far a driver can see to recognize that a pedestrian is walking across a road and all of his citations were referencing daylight illumination. Thankfully, us jurors were able ask questions like "All of your citations are for daylight illumination. You know this case is about an accident at 3am and the streetlights were broken. What's the distance in that case?" And they would answer that question appropriately, but I didn't consider it impeccable character that they had acted all the while up to that point like the numbers they were giving were for the conditions of the case. And yes, they would always say they were familiar with the conditions of the case before answering any questions, so I generally felt is was lying by omission.

defendants should always be able to challenge expert witnesses qualifications.

and how could anyone on hacker news claim there is 'no evidence' that unreviewed software may do something other than what people claim it does? especially if it 170,000 lines of matlab?

from the article:

> "Without scrutinizing its software's source code – a human-made set of instructions that may contain bugs, glitches, and defects – in the context of an adversarial system, no finding that it properly implements the underlying science could realistically be made," the ruling says.

Particularly when TrueAllele specializes in

>probabilistic genotyping [1]

and

>solves mixed DNA samples, low template DNA and kinship problems [1]

And the product sheet discusses

> models for PCR artifacts [2]

This sure feels like they go way beyond typical dna match technology into maximally extracting information via statistics, so their methods definitely should be open for review.

[1] https://www.cybgen.com/products/casework.shtml

[2] https://www.cybgen.com/solutions/brochures/unique_features.p...

Absolutely not! Please, if you want to know why, look into expert witness testimony as a whole, possibly can start with “burn pattern experts”.

Not to mention we have a constitutional right (6th amendment) to face and cross examine witnesses against us in the US, a black box should not void that right.

> I'm not so sure we should be allowing challenges to expert witness, with no evidence

Any challenge is "allowed" in the sense that the parties can try to convince the judge there is a flaw.

The question is one of burden of proof: should the expert prove they have a trustworthy result, or should the expert testimony stand on their credibility alone, and the defense scramble to poke holes into it armed with incomplete information.

For a system that can put people into jail for life and that acts as a direct extension of state power, it's obvious the latter can have disastrous consequences. Effectively, any one who acquirers the "judicial expert" title can put people in jail at their whim.

It seems reasonable to have a lower bar for challenging expert witnesses in criminal cases versus civil ones.

When I was a public defender I made my career by cross-examining expert witnesses. There are a dozen innocent people walking around free today because I challenged the expert despite going into the cross "with no evidence."

What you call abuse, I call due process.

(And your proposed solution is already part of the expert certification and questioning process in criminal trials, so it isn't a solution. Experts must testify that the equipment they used was calibrated/certified/etc based on the standards applicable to the field.)

Testify does not equal independently certified. Also, double-blind testing is not really subject to faking. Its the standard of science.

Responses to my comment are all around 'how it is now' which seems not to be very reliable. A responsible testing/certification process would address that.

You've just moved the goal posts from someone who can be crossed in court to someone who can't, which is worse, because then you have now way of verifying that the "independent" certifier is actually independent or that the certification was properly done.

You still need to allow the "independent certification" to be examined in court to validate both the independence and the certification. The means testimony on the applicable standards, and how the specific equipment used satisfied those standards. And right now, the expert witness already does the latter with respect to calibration, etc., for the equipment they used.

The skepticism has to end somewhere. Else ever court case becomes an infinite regressing into doubting the foundations of everything.

Of course self-certification is not the best. To bring up that straw man (as most of these posts do) is not advancing the conversation.

That straw man is essentially the status quo at the moment...

Reread the comment being responded to? It started with that, suggested an alternative. Anything helpful to add?

Yes, the skepticism has to end somewhere. But when someone is facing years, or life, in prison, that skepticism absolutely should not end before it validates the primary evidence of guilt, and the tools used to purportedly analyze that evidence.

If you can't validate the equipment used to analyze the DNA, gunshot residue, or other scientific evidence, then that evidence should not be allowed in court, because that's prima facie reasonable doubt.

Reply on news.ycombinator.com