FTC orders 'gun detection' tech maker Evolv to stop overstating effectiveness | Hacker News Reader

FTC orders 'gun detection' tech maker Evolv to stop overstating effectiveness

techdirt.com

148 pointsshakna128 comments

Multiple concert venues in my city use these, so I interact with them all the time. They have replaced standard metal detectors, bag searches, and manual patdowns w/ hands and/or metal-detecting wands. Security checkpoints are the primary point of delay for getting into venues, and places that have rolled these out process people through about 95% faster. It's a huge difference. If it does trigger, you just get the manual patdown you would have gotten anyway, so the false positive cases aren't any lost time.

The article and settlement seem to only mention the false positive rate, which is a bad thing to focus on. Every true positive is a much faster experience. Only subjecting 110 out of 3000 people to a longer search is a big improvement. Given the negative outcomes of a gun slipping through and the lack of a cost of a false positive, we probably want it to be tuned to be more false positive prone anyway. We don't need these to detect guns THAT well, we just need them to weed out people who definitely don't have them.

I do have concerns about what its false negative rate is relative to the standard practice it replaces. I do not really trust whatever psuedo-AI they're bolting to their metal detectors; it's probably easier to get a gun through. That said, the false negative rate probably isn't good already. TSA isn't great on their false positive rate, does more intense screening, and isn't being staffed by hungover 20-somethings. So maybe the false negative rate didn't actually increase by much?

>TSA isn't great on their false positive rate, does more intense screening, and isn't being staffed by hungover 20-somethings. So maybe the false negative rate didn't actually increase by much?

TSA is abysmal on the false negative rate for things that actually matter. The FNR for actual weapons and explosives is somewhere between 80 and 95%[1]. It's because they waste all of their attention looking for nail clippers and water bottles.

Even an FNR of 50% would be a massive improvement.

[1] https://abcnews.go.com/US/tsa-fails-tests-latest-undercover-...

An acceptable false negative rate really depends on the consequences of getting caught.

If someone's nefarious plan depends on smuggling a gun in, they want to be confident they won't be arrested or shot at the entrance. Even failing to detect 20% of firearms means there's an 80% chance they'll be caught before they can do whatever it is they plan on doing. This is also why it's important to have armed guards alongside the scanners. Scanners aren't very useful if the only armed person is the bad guy.

If the consequences of getting caught are negligible (as is the case for anyone trying to bring a box cutter through airport security), then the attacker can try as many times as they want without issue. Even if the false negative rate is low, they only have to get lucky once.

Annoyingly, I can't find any info about false positive/negative rates for various scanners. There doesn't seem to be the equivalent of Consumer Reports or Underwriters Labs for scanners. My guess is that the numbers must be pretty bad if companies aren't willing to go through public 3rd party testing.

The TSA's own security tests clearly show a significant percentage of guns, knives and explosives regularly get through. This is further confirmed by the number of travelers who, after arriving, discover the handgun they accidentally left in some pouch in their suitcase that was never detected.

> There doesn't seem to be the equivalent of Consumer Reports or Underwriters Labs for scanners. My guess is that the numbers must be pretty bad if companies aren't willing to go through public 3rd party testing.

Of course they are but the main reason there's no publicly available objective testing isn't only that sellers don't want it. In reality, no stakeholder in the security market wants it. The vast majority of high-volume public security like airports, concerts and sporting events is largely unnecessary and mostly ineffective but our current political/media environment requires appearing to "do something" to "make it safe". The Vice-President of "Make it (Seem) Safe" knows that their shareholders, politicians and the public aren't willing to pay more or be even more inconvenienced than they already are for 800% better "Make it (Seem) Safe"-ness.

Metaphorically speaking, the tiger repellent is working just fine, thank you. Those truly worried about tiger attacks feel safer and those being well-paid for preventing tiger attacks can claim virtually 100% effectiveness. So, if you start the world's best Tiger Repellent Testing Laboratory, you'll find a shocking lack of interest in buying your test data from both sellers and buyers in this brisk, profitable and growing market. Much like the lack of interest in objective testing data for lie detectors, astrology readings and placebo pills. The smaller minority of customers actually willing to pay more for improved detection (like Tel Aviv airport), do their own in-context performance testing anyway. In fact, a good proxy for doing your own effectiveness testing is available for free. Just look at what those under constant active threat with real consequences actually pay for and do.

I unknowingly transported ammo both to and from Mexico. I used an old backpack that I had previously used as a range bag from years ago. I ended up finding several 223 rounds in Mexico, then when I got back, even more 22lr.

Others have not been so lucky

https://www.independent.co.uk/news/world/americas/kyle-busch...

Failing to detect 20% of firearms is probably a really deal. I would bet the majority of findings are just ordinary people who forgot to leave their gun in the car (akin to how TSA used to take grandma's knitting needles away every flight).

So, if your process really only detects people not trying to bypass it and people not even in the wrong then it's a problem.

Although I mean with the long lines at security you might as well just gun everybody down outside the stadium in that nice open area they are all packed into ...

I guess on a more general level, I'm confused why a metal detector with bit some machine learning on top wouldn't make a better widget? You'd think that different shape metal object produce different magnetic flux, and there's probably more dimensions to that than just size?

If it's just a gate that you are walking through then it's all or nothing. Basically just a metal detector.

If you have a conveyor belt system and a CT scanner like setup then yes, you could build a better metal detector.

Theme parks are switching to these too and it's oh so much nicer to just walk through and if you're unlucky having your bag searched.

> Security checkpoints are the primary point of delay for getting into venues, and places that have rolled these out process people through about 95% faster. It's a huge difference.

I assume expediting peak crowd throughput at low labor cost is the primary, if not entire, value of the device. I hate that it's being marketed dishonestly but I also assume most concert venue buyers know (or suspect) it probably doesn't work all that well in practice. However, in a concert context accurate detection isn't their main priority. They need to get more bodies per minute into the venue at lower cost while appearing to conduct security checks sufficiently 'real' enough to act as a deterrent to get those who care about getting 'caught' to leave their knife or concealed carry handgun (or whatever) in the car.

The only hard and fast requirement is meeting the contractual security requirements of the venue and promoter's insurance carriers - because no insurance = no concert. It's a bonus if the 'security' also looks plausible enough to reassure the small fraction of perpetually fearful people statistically challenged enough to actually worry about terrorists or an active shooter killing them while at a Taylor Swift concert (as opposed to the infinitely more likely chance of dying in a car crash on the way to the concert).

In a perfect world, everyone would be rational and numerate enough that we wouldn't need to maintain the pretense of 'security theater' in contexts where actual security isn't necessary. But in the imperfect world we live in, I prefer having concert (and airport) security be as minimally disruptive and inexpensive as possible regardless of effectiveness (since it's unnecessary and mostly ineffective in those contexts anyway). I just wish companies would sell these products as 'security placebos' instead of lying about it because fraud is wrong.

I don't know anything about the devices that are being sold, but it doesn't seem impossible to me that with better signal processing from a metal detector, you could reduce false positives a bit while maintaining or slightly improving the false negative rate.

Sure, I agree that's an interesting and likely solvable technical problem. However, the vast majority of the addressable market in today's over-secured society don't really need improved detection. Concert venues, sports arenas and similar customers buy massive volume and they are much more concerned with faster throughput enabled by shorter cycle time and minimal false positives. Of course the head of security at Madison Square Garden can never publicly admit they don't care about better detection enough to pay more for it, but I'm confident the sales managers at these security vendors understand exactly what their largest market segments really care about.

Customers like Tel Aviv International Airport, who actually care to some meaningful extent about improved detection, are a small minority segment of the overall market. Creating new technical measures able to demonstrate improved performance in rigorous objective tests on the metrics these customers care about (some sweeter spot on the matrix of false pos, false neg, true pos, true neg, net throughput, cost) would be valuable but only to that small segment.

Note that I said lower false positives and equal or slightly better false negatives, which aligns with what you say customers want.

Of course I suspect venues really don't care about false negative rates much at all, so there's a big temptation for everyone to just turn sensitivity down.

Very likely, but who's going to pay for the engineering efforts? If the customer doesn't care about having a better device, or a better device would actually make their job harder, then it's wasted effort on the part of the manufacturer.

Engineering exists to solve a problem. It's entirely likely that your definition of the "problem" differs from that of the paying customer.

> get more bodies per minute into the venue at lower cost while appearing to conduct security checks sufficiently 'real' enough to act as a deterrent

Or they just need to convince most of their customers it will be safe to attend, while covering their ass by following "best practices" if something slips through, people get hurt and they get sued.

> convince most of their customers it will be safe to attend

Apparently, you've never met my Aunt Sue. She has a graduate degree in innumeracy with a minor in illiteracy and a specialization in worrying about whatever the media tells her to worry about. However, she always votes.

More seriously, it's not cost-effective to "convince most customers it will be safe enough to attend." The game theory around fallacious public perception makes it a losing proposition for a politician or company to ever appear to reduce security requirements because as soon as "rare bad thing happens", they will be blamed - even though their reduction in pointless measures had no bearing on it.

Most independent experts agree that securing cockpit doors in 2002 made subjecting every passenger to the TSA's increased security measures unnecessary and, objectively, a very poor ROI in both cost and disruption. However, the TSA will never, ever go away - even though it could and should. Not only is reducing security politically costly, the TSA is now a multi-billion dollar federal bureaucracy, paying hundreds of vendors with lobbyists and employing tens of thousands of unionized workers spread across the most populous congressional districts. Yes, this is frustrating.

They don't need to convince most of the public, just enough so they can fill up their venue. There are some Aunt Sues who are too afraid to go to concerts, but concert venues are still able to fill up when they have a popular act so it stands to reason that they're managing public perception of the risk well enough for their own needs.

> The only hard and fast requirement is meeting the contractual security requirements of the venue and promoter's insurance carriers

I think it would be a good idea to create an explicit carve out in the law saying that there is no premise liability for a property owner or event organizer due to a third party committing a crime.

So, do away with all negligent security cases?

Yes. In general, business owners aren't expected to prevent crimes against their customers. If someone attacks me at a bar or grocery store, I probably won't get very far trying to sue the owner for failing to check everyone for weapons on entry. I'm not sure I'd have more success with a concert venue, but it appears insurance companies perceive enough risk to demand certain procedures.

Codifying that expectation in law would reduce costly and obnoxious security theater. Of course, a business advertising a certain level of security could be sued for failing to provide it.

Ok, but it seems like a bit of a non-sequitur to say “ business owners aren't expected to prevent crimes against their customers” when there’s a body of law to the contrary.

Is there? In most US states, the concept of premises liability seems to be derived entirely from case law, not statute. Some states appear to have statutes limiting its scope, such as https://colorado.public.law/statutes/crs_13-21-115

Edit: to be clear, I don't think there's anything actually stopping someone from attempting to sue a bar or grocery store over a crime committed there, but it usually doesn't happen and would likely be an uphill battle for the plaintiff.

So what? It's not like common law has less effect. "Body of law" is understood by lawyers to include both common and statutory law.

The point is that it can easily be overridden with statute

"business owners meeting definition X are only liable in conditions Y"

This is a stupid point. “The current law can easily be overridden by passing a new law.”

No shit.

I think a better argument here is that common law/case law here is ambiguous enough to create a situation where there's an unreasonable and unpredictable risk for certain kinds of businesses.

Another problematic case this sort of liability leads to is hotels in Las Vegas routinely searching guest rooms after a lawsuit following the 2017 shooting. I don't think it's desirable to expect hotels to search rooms or to call the police if someone has "too much" luggage. That's paranoid, an invasion of privacy, and unlikely to prevent a future mass murder.

I don't want a world where I have to submit to searches to go anywhere or do anything, and I hope that's not a fringe position.

You say that as if statues are any less ambiguous.

I was only responding to GP's statement that common law can be overridden by statute.

If I wanted to respond to the idea that premises liability should be eliminated then I would have responded to your first post.

And I actually do think that most people would call your position a fringe position once you actually start talking details like "but what about guns in schools?" If you truly believe that you shouldn't have to submit to a search to go ANYWHERE or do ANYTHING then you hold a fringe position.

My position is that it is undesirable for premises liability to lead to an increase in private businesses searching their guests, not that nobody should be searched anywhere for any reason. I do think it should be rare in practice: airports, probably; concerts, probably not absent some unusual threat; hotel rooms pretty much never.

It appears routinely searching students in public schools is fairly rare in the USA; under 8% use metal detectors[0]. That certainly does not mean they're allowed to bring guns, just that they probably won't be discovered if they do.

[0] https://nces.ed.gov/fastfacts/display.asp?id=334

No, your position is: "I think it would be a good idea to create an explicit carve out in the law saying that there is no premise liability for a property owner or event organizer due to a third party committing a crime." and "I don't want a world where I have to submit to searches to go anywhere or do anything."

This means that nobody should be searched when they go anywhere or do anything, and if they aren't and someone gets shot by a third party, stabbed by a third party, or mugged by a third party then there is no liability to the business/landowner in any case. Ever. Searches will not be rare in practice, they will not occur. Airports? Never. Concerts? Never. Hotel rooms? Never. Schools? Never.

And hence we've already established the problem with your position and why it's fringe. Not even you can realistically argue for your own positions without caveats. This is a great example of a motte and bailey fallacy.

I see the miscommunication now. It's possible to interpret what I wrote as "no places/activities should require searches", but what I meant was "not all places/activities should require searches". That's a bit hyperbolic of course since the resources don't exist to search everyone, everywhere, all the time. There has been an increase in recent years, and I would like it reversed.

I am opposed to premises liability being a motivation for anyone to conduct searches. Liability isn't the reason searches are conducted at airports or courthouses to give a couple examples, so eliminating it would not eliminate those searches. Businesses also might have other motivations, such as making their customers feel safer; if that outweighs customers finding it annoying or offensive, some of those would likely continue.

> someone gets shot by a third party, stabbed by a third party, or mugged by a third party then there is no liability to the business/landowner in any case. Ever.

This does correctly state my position.

I will concede the technical point: there is a body of law that sometimes expects business owners to prevent crimes and sometimes doesn't, with a whole lot of ambiguity about exactly what any given owner is actually expected to do. I think that ambiguity should be reduced by putting criminal acts out of scope.

Interesting how a guy whose entire administration is being investigated and indicted by the FBI ran almost entirely on a ‘law and order’ platform, with this pilot being one of the staples.

The "investigated and indicted by the FBI" that lead to nothing for the "law and order" candidate? Interesting that the "lawfair" candidate would have had the complete opposite outcome, had the laws been equally enforced.

You're confusing Eric Adams with Donald Trump.

Clark County School District recently signed a $5 million contract with a different AI “Weapon Detection System”. There is no way any of them are real is there?

Edit: CCSD uses Remark. Maybe their AI actually knows what a person walking with a gun somewhere on their body looks like in all situations and for all skin colors.

Seems like it would cost less than $5m to independently test this unit before purchase.

There’s definitely some form of filtering you can apply to metal detector results to make a relatively good guess on whether it’s a false positive or not.

Just need to train it with a variety of hidden and unhidden items. I kinda like the idea of a team of test people hiding their weapons everywhere and then taking multiple trips through the thing just to teach it.

It’s not a metal detector it’s optical. What does a person with a pistol strapped to their inner thigh walk like? Or a knife? There is no possible way to detect that visually.

> There is no possible way to detect that visually.

Well, let's not go that far. Security personnel definitely detect people carrying weapons by gait and the way their clothing hangs. Certainly computers could help do the same.

Can it ever be reliable enough? Ehh, I doubt it.

And will that disproportionately flag against the disabled? Like people who drag a leg, use walking sticks and frames, etc? Have an insulin pump under the clothing?

Unfortunately, security screening already pays disproportional attention to the disabled. E.g., we are talking about this in a thread about metal detectors, which will flag on most of these people.

I cannot find any summaries for the cost of these detection devices. What did this cost, anyone know?

Probably a lot.

https://en.wikipedia.org/wiki/ADE_651 made tens of millions of dollars at ~$5,000/unit.

> The laboratory found that the card contained only a standard radio frequency ID tag of the type used in stores to prevent shoplifting. According to the laboratory's Dr. Markus Kuhn, it was "impossible" for the card to detect anything and it had "absolutely nothing to do with the detection of TNT". The card could not be programmed, had no memory, no microprocessor and no form of information could be stored on it. Despite the high cost of the devices, the cards were worth only about two to three pence (3–5¢) each. Kuhn commented: "These are the cheapest bit of electronics that you can get that look vaguely electronic and are sufficiently flat to fit inside a card." The "card reader" was found to be an empty plastic box.

> He told The Times that ATSC had been dealing with doubters for ten years and that the device was merely being criticised because of its "primitive" appearance. He said: "We are working on a new model that has flashing lights".

Damn, why am I not a scam artist? This worked so well for so long. If he put $20 of random electronics and tiny glass tubes of chemicals in there I think he could've gotten away with it.

> Promotional material issued by ATSC claimed that the ADE 651 could detect such item as guns, ammunition, drugs, truffles, human bodies, contraband ivory and bank notes at distances of up to 1 kilometre (0.62 mi), underground, through walls, underwater or even from aircraft at an altitude of up to 5 kilometres (3.1 mi).

Could they have at least tried to make their lies believable?

I assume it's like Nigerian Prince emails - if they were believable, they might have appealed to buyers who would have done something like purchasing one and testing it, instead of spending millions of dollars on garbage.

This is it exactly. All of the serious people left before they finished the sales pitch. If they had sold one to someone competent that would be grounds for a lawsuit or fraud charges. They had to filter out the customer base to just two kinds of buyers:

1. The people who fail to realize that it's a big scam and will be fat dumb and happy forever.

2. The people who don't want something functional, they want a "probable cause" generator they can pull out when there isn't any evidence to go on.

Sounds like they're basically making minimally plausible props for customers more focused on security theater performances than any actual security.

It's scummy and dumb but legally the only real problem I see is lying and collecting money under false pretenses making it fraud. Alternatively, they could have chosen to market these products confidentially as essentially security placebos. There's a market for things like fake security cameras, which arguably have some value for deterrence and reassurance. I suspect many of their current customers would probably have been just as happy buying these products knowing they didn't work.

I don't think it's quite as innocent as security theater. I think these are more for generating probable cause when they need to shake down someone who isn't doing anything illegal.

Ah, well if the design allows manual triggering as opposed to some random percentage, then that would obviously be bad.

It’s mostly based on the ideomotor effect. It’s all manual triggering, just mostly unconscious.

"Hear me out, we run the same playbook, but this time with AI"

https://en.wikipedia.org/wiki/ADE_651

>The ADE 651 is a fraudulent bomb detector[1] produced by the British company Advanced Tactical Security & Communications Ltd (ATSC). It was claimed to detect many substances, such as drugs or explosives, from long distances. The device was sold to various countries, particularly in Iraq where the government was claimed to have spent £52 million for security operations.

>The device features a swiveling antenna attached to a plastic grip and requires charging by a user's static electricity. Users would insert "programmed substance detection cards" to supposedly detect specific substances, which were claimed to absorb the vapors of those substances. However, investigations revealed that the product was incapable of detecting anything, essentially being a dowsing rod. The ADE 651 was used primarily by Iraqi security forces for security checkpoints. Due to the false sense of security, many critics pointed to numerous incidents where bombings occurred despite the presence of the ADE 651 at security checkpoints, underscoring its ineffectiveness.

I mean, the company has never really said how their systems worked. It's basically left the users/scanned with the impression that it's magic. It seems to me that it's basically some of 3D metal detection paired with cameras. It would be cool to understand how they make it work, but I guess it's a trade secret. They haven't readily put up numbers about false positive rates, and even less data about false negatives. I get the impression that this relies heavily on placebo effect - "they have scanners so I better not try to take my weapon in".

I thought advertising and marketing materials were expected to exaggerate things to the point of blatant falsehood.

Is that changing, or is this company being singled out for some reason, or are they really that much worse than everyone else?

It is, and has long been illegal to make false statements of fact in marketing materials.

Puffery, on the other hand is allowed. That usually entails non-falsifiable statements like "Evolv is the best way to detect guns". "Best" doesn't really mean anything because there are a bunch of tradeoffs that go into designing a security screening system.

Right. "The world's best cookie". I think the idea is that no reasonable person would understand that statement to mean that literally, of all the cookies in the world, this one right here is the best.

At least equally important to whether a reasonable person would take it literally is that there's no way to prove it isn't. If I call something the best cookie, the closest thing that has to a concrete meaning is "I like it better than any other cookie".

In the US (YMMV) there's a distinction in advertising between exaggeration for effect, like "this expensive cream will make you beautiful", and straight up falsification like "we can detect all guns everywhere" when you demonstrably can't.

It’s illegal, and has been for many decades. It’s not enforced nearly as often or as strictly as it should be.

Reply on news.ycombinator.com