Hacker News Reader

Comment by woah | Hacker News Reader

Sounds like it’s well-intentioned, but I’m afraid that given the incentives of all parties involved here, and the complexity of the technology, there will be a lot of false positives.

It’s obvious that the police will want a device that produces more convictions, no point in disputing that.

I’m wondering if you can elaborate on what incentives your company has to produce accurate devices, against your customer’s unstated preferences. What’s keeping you in check? Are you worried about class action lawsuits from victims of false positives, if it were independently proven that your device is overly sensitive?

View on HN

You raise a good point. Interestingly, your comments run contrary to what a lot of law enforcement officers have said to us.

In general, the police are hyper vigilant about buying only devices that are independently validated to be very accurate. Every conversation we have had has eventually lead to "is it NHTSA approved?". The reason for this need for third party validation is that the police are incredibly court room sensitive. If there is any chance a defense attorney would be able to pull out a study showing low specificity or sensitivity for a device, the police will simply not buy it. Third party validation gives them that guarantee.

You are right that the police want more convictions (or less time consuming convictions), but the way they do that is by having very accurate devices that are defensible in court, not inaccurate devices that risk cases being thrown out. That is why making sure our device has low false positive is very important.

I admire what you're doing, and hope you can adhere to scientific integrity while developing and selling your product, but that simply isn't true. At least in Washington state. A few years ago Washington State purchased a whole bunch of Draeger 9510 (alcohol breathalyzer) units without ever having tested them. After two years of sitting unused, the warranties were going to expire and they finally got around to testing them, and purchasing extended warranties at that point. However their initial tests failed multiple accuracy tests to the point that the state's lab halted testing, since it was so off. A few months later Draeger patched the firmware and OS application (the breath/alcohol measurement firmware being written by a single contractor in Texas, and versioned as 0.7) and while testing looked better, it still wasn't accurate and they just pushed it out the door for deployment...they had already bought them, after all. I'll link the records below...they didn't even have the correct versions of the software and firmware accounted for and signed off, which led to other problems (and Draegar has had other problems of accuracy in other court cases, such as Florida, but that's a side tangent).

My point is, a lot of state governments (at least in the US) spend money first and verify later, and 'verify' can be a damnably loose term. Loose enough to violate WAC 448-15-020 (in Washington state's case) which requires a 'reasonable degree of scientific accuracy'.

EDIT: Apparently the original link (below) is no longer available in WSP's records, which is interesting. You can view the document via archive.org, here

https://web.archive.org/web/20170211114622/http://www.wsp.wa...

original (defunct) http://www.wsp.wa.gov/breathtest/docs/webdms/Draeger/Valid_D...

Thanks for the info. Definitely something we have not run into or considered yet. I guess we just naively assumed that all police precincts were as scrupulous as those we talked to :(.

There is some stuff we can do, however. For instance, we can directly build the QC schedule into the device so it does not work unless you get it calibrated. We can also have QC cartridges that have a barcode that the device reads to make sure you actually ran a calibration check properly and the device passed.

We were so focused on internal methods for making the device accurate, but maybe we should put some thought into how we can make sure the police use it properly.

Really, if you're producing something that is reliably accurate, I would expect that would be all anyone would expect, along with proper training, verified updates, and audited software/firmware, etc. In the case I cited above it was more an issue with the lack of accuracy of the product, and sloppy follow through from the vendor, with money and political pressure thrown at the problem - something which the police departments themselves likely have less influence on anyway.

Everyone says they are ethical including the unethical I'm not sure how you possibly took this at face value.

>I guess we just naively assumed that all police precincts were as scrupulous as those we talked to :(.

Bless your heart.

If your product is at all worthwhile, it will constantly be tossed back and forth between lawyers, trying to uphold or disprove its validity. ("Well here's an independent study done on how arresting officers with shaky hands get inconsistent results when administering the SannTek Cannibus test" et al et al ad nauseum) While yes, this is a problem in any society that applies capitalism principles to the practice of law.... I don't think there's necessarily a better system, sadly. Don't feel responsible to fix it all yourself. Anyway, pursue the technology as well as possible, by all means work in as many checks and balances as possible for correct use, and constant calibration, BUT ALSO talk and document openly during your design phase and testing the ideology you develop to govern false positives versus false negatives. This is an extremely engaging technology ethics discussion. If you drive the error to be false negatives, at a rate of 5%, will you not be able to sell it, because your customer is most interested in an effective tool? What is the societal benefit of getting this device to be accepted and used by the police? If you can achieve FN=4% is it worth it, or is the societal benefit actually much higher, possibly at FN =20%. Can you just run the test 5 times in a row, or with 2 stand alone devices to reach the dependability rate you desire? If you drive the error to be false positives, at a rate of 1%, can you sleep at night with the lives your product will effect, or do you need a fall back like your device determining with 100% accuracy that cannibus is present in someone's system (just not a definitive, exact amount), PLUS a statement from an arresting officer stating erratic or dangerous driving or behavior in conjunction with your device's input. I'm very excited about what you're doing. I didn't know (still not yet convinced) that the science is solvent, but I've been asking about this device for years.

> In general, the police are hyper vigilant about buying only devices that are independently validated to be very accurate.

How do you reconcile that with the reality of the NIK tests which are notoriously unreliable and ubiquitous among police departments?

Like the lie detector?

> It’s obvious that the police will want a device that produces more convictions, no point in disputing that.

Is that obvious? If this were as overwhelming an incentive as you say, we wouldn't have accurate breathalyzers for alcohol. But we do, and it's simple to see why. 1) Cops aren't the only customers; medical establishments, individuals, and workplaces also have a legitimate interest in measuring impairment. 2) Tech like this doesn't remain novel forever, it can be checked for accuracy, and if you made a device that exaggerated impairment your reputation would tank, your company name would become a political hot potato, and your sales would never materialize. 3) There is definitely a point in disputing the idea that cops are indiscriminate gangsters roaming the streets trying to lock up any citizen they don't like the look of. Police have their personal and institutional biases, but by and large, most are interested in tools that allow them to safely and fairly administrate the rule of law. Police don't have much career or institutional incentive to arrest someone for a DUI at a traffic stop or not—they just stop bad drivers and check. It's prosecutors who are trying to rack up conviction numbers as a matter of career advancement and political viability. But even that gets checked: their work will be subject to an adversarial legal system, at which point any evidence that the device doesn't work would come out in court and tank their conviction rate.

So... where do you see perverse incentives here, again?

Do we have accurate breathalyzers for alcohol? AFAIK, the manufacturers aggressively resist requirements to publish their source code or calibrations for testing. Also, AIUI, the devices require recalibration for accuracy, and it's not clear that police stations perform such maintenance.

No idea how rampant this is, every system will have flaws. But I question whether we can use the alcohol breathalyzer example without acknowledging there are significant unanswered questions w/r/t accuracy.

If breathalyzers were inaccurate, uncalibrated, or unmaintained it would be trivial to get any conviction relying on their measurements thrown out.

What makes you say that?

Only if you're allowed to test the devices in question, and the courts believe your experts over the manufacturers.

This doesn't prove my point, but manufacturers do aggressively fight against researchers identifying flaws in their products. Given the high stakes (+ the fact this isn't a security issue, really), it seems like a major smell to me.

There are reports out there of people convicted, who are blocked from examining the devices used as the primary piece of evidence against them.

Also, after years of effort to discredit breathalyzer results, convictions ARE being overturned regularly.

NJ overturns > 27k convictions because of breathalyzer issues - https://www.nytimes.com/2018/11/13/nyregion/nj-dwi-convictio...

MS judge suspends admission of breathalyzer evidence -- https://www.bostonherald.com/2019/01/10/breathalyzers-tempor...

More relevant info - https://www.zdnet.com/article/draeger-breathalyzer-breath-te...

Nothing about the court system is trivial.

Mind you, that's not unprecedented. https://arstechnica.com/tech-policy/2009/05/buggy-breathalyz...

A friend of a friend works at a facility which calibrates + certifies police breathalysers and by his account below 0,015% BAC the result is garbage, so some countries choose to set the limit at 0,02% which is enough for the devices to indicate properly most of the time.

The National Highway Traffic Safety Administration report to Congress [0] seems pretty clear in terms of the art of the science and the impairment observed from national data.

I would avoid incarceration of anyone from a test like this without significant and clear data that there is an actual problem we need to resolve. There are plenty of well proven driving problems we could focus on correcting.

[0] https://www.nhtsa.gov/sites/nhtsa.dot.gov/files/documents/81...

> What’s keeping you in check?

It's going to be fairly easy to test this. You take one group of people who haven't smoked in the last few hours and you test them. Make sure it doesn't go off. You take another group who have smoked (I'm sure plenty of college students would be happy to sign up for this trial) and you test them, make sure it does go off. It's pretty simple. It's a machine with an empirical number, so you don't even have to do this double blind.

And if it doesn't give good results? It won't hold up in court. Every lawyer defending someone with a positive result (whether false positive or true positive) will argue that the device isn't accurate and the case gets tossed (or even the ability to use these as evidence). If it doesn't hold up in court, police really won't have much incentive to use these anyway.

It is not that simple. Within the control group of non-smokers you need people who smoked yesterday or 12 hours before to determine if the testing is successful.

But even that is too simplist. You need to activity give a specific dose to a group with similiar profiles. You need to test with different profile groups.

Remember we are testing currently high not has been high.

The basic functionality of a breathalyzer is to take a given volume of an exhale and measure the concentration of a given particle as an indicator of intoxication. So long as an individual's results are greater than or equal to 0.08 then it becomes an unacceptable risk and they are considered intoxicated. If this process is going to test in a similar way, there is no need to test specific doses or profiles because the concentration isn't variable, it's a static amount. With alcohol, we don't consider weight, or what they had to drink, or how long they've drank so why should this be any different? All that needs to be defined is what the law considers a concentration which should represent a level where the risk reaches an unacceptable percentage.

I think maybe you misunderstood @wolco's point? I think what they are saying is that, for example, you might do all of your testing on college age males and find that it doesn't give any false positives for people who smoked yesterday but not today. But you still need to test it on 60 year old women to make sure it doesn't give false positives there. In other words, you need to test it for varying lengths of time and also to make sure that it has consistently acceptable results by race, sex, weight, age, etc.

If that is indeed what they're saying, my point still stands. The blood alcohol limit has no variations for age, sex, gender, race, weight, time since last usage, or any other factor. It just goes by concentration. My point still stands. What the law considers the concentration which indicates intoxication is a metric based on what is considered an acceptable level of risk.

Additionally you need to test it doesn't trigger a positive when someone had a cinnamon flavored mint before. Or a citrus one. Or ate a mushroom sandwich. Or has diabetes..etc etc

Even alcohol breathalyzers are regularly miscalibrated or defective and found to produce false positives. Legal pressure from manufacturers keeps this quiet.

https://www.zdnet.com/article/draeger-breathalyzer-breath-te...

That is why many states follow a positive breathalyzer test with a blood test. In my state, you can refuse them both, but you lose your license temporarily for doing so. I think that's fair. I understand some are skeptical about giving blood to the police, but if you can't prove you're not drunk or high after a positive reading on a less accurate device, then you aren't allowed on the road. The risk is too great for everyone else.

In Texas, you receive a mandatory suspension for refusing just a breathalyzer, even if you are found to not be over the legal limit, or are never convicted of a DWI. Blood tests are then often obtained via warrant (and it's typical to have a judge "on call" to sign and fax those orders on demand)

That's utterly disgusting that the state can invade your body and take blood out with a warrant.

You agree to these things when getting a license. It's like the EULA we never read.

I personally think it's utterly disgusting that people are allowed to operate what are essentially giant weapons in public with extremely minimal testing or training, and with minimal limits on alcohol usage. I think it's totally reasonable to ask people to give up some of the rights they'd normally have, when those people want to operate heavy machinery in public.

My understanding is that GA pilots are not allowed to fly within 8 hours of consuming any alcohol (my understanding is that most commercial outfits have more stringent limits) - I personally think this makes a lot of sense for operating a motor vehicle, too.

It takes between 1 - 3 hours for a single mixed drink or pint of beer to be metabolized.

As a starting point I think you will agree that someone who has a BAC of zero ought to be able to drive.

The majority of drunk driving accidents are the result of people that are plowed. More disturbing by far those who ARE caught and jailed have driven hundreds of thousands of miles drunk as a skunk before actually getting taken off the road.

As BAC decreases accident rates decline towards normalcy at around 0.06 its difficult to distinguish the accident rate of those who have consumed and those who have not.

People at that point neither walk nor drive funny and are very unlikely to be discovered unless you stop and test everyone or you happen to be black or Hispanic.

Instead of tightening already reasonable standards to ridiculous levels we would save vastly more lives by actually enforcing the laws we have effectively.

>As a starting point I think you will agree that someone who has a BAC of zero ought to be able to drive.

Not if they are tired or hungover. I mean, I don't think people should drive when they are tired for other reasons, either. It's a dangerous thing, and something you should only do when you are at your best.

Alcohol is just an easy target (and an important target) because it both degrades your actual ability, while at the same time, increasing your opinion of your ability. Because of that second effect, it's especially difficult to tell when you have had too much to drive... because one of the common side effects of alcohol is an increase in confidence.

It should be difficult to drive, and it should be easy to lose the ability to drive.

>Instead of tightening already reasonable standards to ridiculous levels we would save vastly more lives by actually enforcing the laws we have effectively.

eh, we already have rules against driving tired. they just lack an objective test.

I think that if you think you can operate a motor vehicle or a gun in public at 0.06, you are not being nearly careful enough. (And I suspect the additional confidence that alcohol gives you might actually contribute much to accident rates. I know I certainly feel more confident one drink in, and confidence is deadly when you are operating weapons or heavy machinery around other people.)

If the device is wildly inaccurate why would you be obliged to prove your innocence barring other evidence of impairment.

It points very strongly toward intoxication. There is just enough false positives that they are required by the state to also take a blood test.

It's a confirmation, and when you're about to sentence someone on the spot with what is often a life changing, multi year disqualification from driving, the courts require the police to be absolutely sure.

You're usually free to refuse the blood test, but the DMV is also free to revoke your license arbitarily.

"It’s obvious that the police will want a device that produces more convictions, no point in disputing that." Uh, why couldn't we talk about it as adults? OP's answer below is excellent. Police want a device that will make their job most effective and profitable, with the lowest margin not only for legal repercussions, but also the lowest possibility of people like you deciding that all police officers are pure evil who just want as many convictions as possible. Doesn't everybody want their job to be like that? BLM has made some good progress (and more is needed yet) scaring some pure evil people who happen to be police officers into wanting what's good and right. Probably not prudent to worry about a new technology that might be oversensitive, especially when the first time this thing is used in a conviction by a person with a rich dad, they'll spend as much as possible throwing doubt on the research supporting it, or how the officer was trained to use it, or etc etc etc, in order to build plausible deniability in court and get the case thrown out.

> It’s obvious that the police will want a device that produces more convictions

If a defendant ever challenges the use of the device, an inaccurate (particularly, false-positive-producing) device is more likely to be thrown out discrediting the device class, not just the individual case, and potentially forcing after-the-fact dismissal of prior convictions.

So, avoidable false-positives isn't necessarily an approach consistent with maximizing convictions.

Unfortunately, it doesn't seem to be that simple in practice; this comment by edmundsato [1] has a lot of good info.

After-the-fact dismissal of convictions may seem like things eventually work out, but remember that in the meantime innocent people's lives are upended as they're thrown into the justice system.

[1] https://news.ycombinator.com/item?id=20718599

> After-the-fact dismissal of convictions may seem like things eventually work out

No, it doesn't.

But, if the police motivation is maximizing convictions that stick, it's a potentiality that reduces the degree to which a system that produces false positives meets that goal. (Unless you have a system that is as comprehensive a conspiracy as the FBI crime lab wholesale fabrication of the entire field of fiber analysis where essentially all the “experts” in the field were part of the scam...and even that eventually collapsed.)

My point is that before these scams "eventually collapse", people's real lives are caught in the crossfire.

It doesn't even have to be a big conspiracy theory. Here is an instance of a breathalyzer manufacturer pressuring researchers to stop researching whether their devices are faulty [1]. Are they actually delivering false positives? Who knows! We need to bake oversight and accountability in from the jump, not try to back our way in once too many people's lives are disrupted.

[1] https://www.zdnet.com/article/draeger-breathalyzer-breath-te...

> My point is that before these scams "eventually collapse", people's real lives are caught in the crossfire.

Yeah, I’m not disagreeing with that, nor does it seem to disagree with my point, that the consequences of the nearly-inevitable collapse makes it dubious that maximizing positive results by encouraging false positives is actually a means to the proposed police goal of maximizing convictions, if by that one means convictions that stick.

I never suggested, and also explicitly rejected, the idea that this even superficially results in a no-harm situation.

I understand. One point of disagreement that I could have made more clear is that I don't think the collapse is "nearly-inevitable" at all.

I'm also skeptical that it provides sufficient motivation to spend the money to bring the false positive rate down to X acceptable level. This is a for-profit company; they're going to try to get the lowest ratio of R&D spend to revenue that they can.

It makes me wonder how breathalyzers work. If you fail, I assume the police still need to analyze a urine sample to prevent false positives. But probable cause is established with a breathalyzer failure.

It depends on the jurisdiction. In general, the police begin with a roadside test. This is either an standard field sobriety test (walking in a straight line, touching your nose, etc) or a screening breathalyzer. If the person fails, they are arrested and taken to the station for further testing. This is either performed using a more accurate breathalyzer, or a blood test. In general, suspects can choose which test they want (blood or breath).

Reply on news.ycombinator.com