Hacker News Reader

Comment by djsumdog | Hacker News Reader

Millimeter-wave machines at airports today are totally and completely useless. Germany's transportation administration reported a 54% false positive rate for these pieces of rubbish. You are better off randomly searching people by flipping a coin.

These are multi-million dollar security theater failures and it bothers me more that people are considering buying even more derivatives based on this garbage technology for anything.

Police officers or security guards could just literally point one at anyone they find suspicious and use it to justify an illegal search, because it will most likely read yes, even if the person's pockets are totally empty.

View on HN

> Millimeter-wave machines at airports today are totally and completely useless. Germany's transportation administration reported a 54% false positive rate for these pieces of rubbish. You are better off randomly searching people by flipping a coin.

Edit: note that it seems I had a wrong understanding of the term false positive rate, ignore this comment.

Statistics doesn't work like that. The false positive rate is pretty meaningless in this context. It only determines how much useless manual work is performed after the machine flags you incorrectly.

You should look at the false negative rate in order to determine how much of a security plus they bring.

That being said, I think the whole airport theater is ridicolous. If terrorists wanted to kill people, they literally have thousands of alternative ways to achieve that.

In this context of using them in public spaces it’s the false positive that matters most. The complaint is that the machines basically always alarm, which means the only thing they can be used for is making up a bogus excuse to manually search / harass someone based on no real suspicion.

But they don't always alarm according to the number that was presented. It alarms twice as many times as it should. That's not good and I certainly wouldn't want these in public spaces.

But you can't just look at the false positive rate and how much more work and unpleasant/unnecessary searches they cause. You have to take into account the false negative rate to get a complete picture. Manufacturers and the people who want these machines will always err on the side of too many false positives than too many false negatives.

Again, I personally think that these machines have no place in public spaces, no matter their hit/miss-rate.

> It alarms twice as many times as it should.

That's not what a false positive rate is. It alarms on half of everyone that goes through. It's utter garbage as a first-line test.

I think this is a misunderstanding. From Wikipedia [1]:

> A false positive error, or in short a false positive, commonly called a "false alarm", is a result that indicates a given condition exists, when it does not.

So a 50% false positive means also that in 50% of alarms the condition was actually true.

What you mean is a 50% alarm/trigger rate. Which, depending on how many people do have something on them that should trigger the alarm, would result in a far far higher false positive rate.

1: https://en.wikipedia.org/wiki/False_positives_and_false_nega...

> What you mean is a 50% alarm/trigger rate.

No I don't. It's just that the true positive rate is close enough to zero that both false positive and alarm rate are roughly the same number.

Nevertheless you're still not understanding "false positive" correctly. If you correctly pick 5 out of 100 true positives without any flaws, and also have a 54% false positive rate, you're grabbing 51 false positives, and >90% of your picks were incorrect.

And I'm pretty sure the true positive rate is well under 5% to start with, which just makes it worse.

"In Germany, the false positive rate was 54 percent, meaning that every other person who went through the scanner had to undergo at least a limited pat-down that found nothing."

This is wrong. Even your quote shows this. A false positive rate is how often it triggers when the true result is false.

> A false positive error, or in short a false positive, commonly called a "false alarm", is a result that indicates a given condition exists, when it does not.

So 50% false positive rate means 50% of people who should not be flagged are flagged.

https://en.wikipedia.org/wiki/False_positive_rate

It is quite definitly true that every second person needs to be patted down without a reason, purely on my personal experience as a frequent flyer

Not quite true. A high false positive rate leads the people running the machines to stop reacting seriously to alerts, which means that they don’t do the manual checks with as much care. This is even more of an issue when everyone in line knows they are bullshit and has no respect for the people running them.

A false positive rate of close to 50% means that every second alert indeed pointed at something real. That should not cause the staff to get unmotivated. Again, without the false negative rate we can't judge if they are complete bullshit or not. If it were a 90% false positive rate then sure, that would clearly have the effects you outlined. I don't think at around 50% that's the case yet.

> A false positive rate of close to 50% means that every second alert indeed pointed at something real.

No it doesn't.

Please see my other comment which references the definition of false positive rate because I think your understanding of the term is wrong. You have a 50% alarm/trigger rate in mind. And please try to reply with a bit more substance than "No".

To quote an article on the subject: "meaning that every other person who went through the scanner had to undergo at least a limited pat-down that found nothing"

> And please try to reply with a bit more substance than "No".

Eh. I was hoping you'd just delete the comment rather than have two parallel arguments.

> The false positive rate is calculated as the ratio between the number of negative events wrongly categorized as positive (false positives) and the total number of actual negative events (regardless of classification).

https://en.m.wikipedia.org/wiki/False_positive_rate

54% FPR means half of all people who aren't trying to bring anything illegal onto the plane gets stopped.

Since basically nobody are actually trying to do this, that also means that more than half of all people going through gets stopped.

The ratio of true positives to positives must thus also be vanishingly small.

No, it means that about 50% of people scanned would trigger a "positive". The false positive rate is the number of false positives divided by the number of actual negatives.

100 people, 50 false alarms, 1 actual problem alerted and 49 marked as safe correctly = 50/(50+49) = ~50%.

That's just wrong. This is ML 101

False positive = (Machine says positive but grand truth is negative) / (Machine says is positive)

In English, a false positive is when the machine declares something as positive but it is actually negative.

This fits a sanity test because ~50% of people that pass through a detector do not get patted down. The truth is closer to 10%. Even then, a true positive probably includes someone with two pennies in their pocket.

https://en.m.wikipedia.org/wiki/False_positive_rate

In statistics and ML it’s (FP) / (FP + TN) aka (false positives) / (actual negatives samples). This is by far the most common definition.

Here ML using the common definition see fall-out: https://en.m.wikipedia.org/wiki/Precision_and_recall

You are right this is ML 101, but you are wrong. Someone else has linked the wiki page, and I've posted it elsewhere and I would strongly recommend you read it as your interpretation is very incorrect.

> In English, a false positive is when the machine declares something as positive but it is actually negative.

Yes, and the rate is based on the frequency this happens in your negative set.

>This fits a sanity test because ~50% of people that pass through a detector do not get patted down.

The 50% figure may be wrong, but my definition of what a false positive rate is not.

You're thinking of FDR: False discovery rate. It equals Σ False positive/Σ Predicted condition positive.

See https://en.m.wikipedia.org/wiki/Precision_and_recall for a table of different terms.

Wow, that definition is counter-intuitive. Ive expected the false-positive-rate to be the number of false positives divided by the number of all positives. Isn't it how "rates" are usually defined? No wonder everybody is confused.

I think it's a perspective thing. You are looking at the rate of false results, so given a particular type of input what rate do you get it right? For negative ground truth your possible outputs are "false (positive)" and "true (negative)".

I agree it can be confusing, but I think people would also make the opposite mistake if it were defined differently.

You're thinking of FDR: False discovery rate. It equals Σ False positive/Σ Predicted condition positive.

See https://en.m.wikipedia.org/wiki/Precision_and_recall for a table of different terms.

This is what they do in Mexico when entering the country from the perspective of flipping a coin. I took my vacation in Maya Riveria and upon entering the country they have a box with a button. You push the button and it lights up green or red. Green means you go, red means you have some additional security checks.

Having a high false positive rate isn't really the end of the world for security screening though, as long as they have a low false negative rate, which I don't see many articles complaining. Even at 50% false positive rate, it's 50% time saved compared to patting down every one. Additionally, these machines highlight potential areas to search more carefully, so there's probably even more time saved there.

If your antivirus or car alarm start flooding you with fake warnings you will ignore then more often than not. It’s human nature and you can’t really change it without some drastic measures like severe punishment or constant checks to keep people on their toes. And this last one just kicks the ball further up the ladder but still to a human with the same predisposition.

Not to mention the culture of suspicion and insecurity it builds up. It's insane, and terrible for democracy.

Having just gone through the airport yesterday I definitely relate to this.

Even when walking through security totally clean you feel helpless, it is probably the closest we come to feeling the pressure of a totalitarian govt. in the USA, it makes me appreciate the bill of rights and not take for granted what it gives us.

I wonder how well a tool would work that anonymises people - so those with power won't consciously, or subconciously, target people with brown skin and large beards - while still showing how they are behaving. You could even use cameras to monitor heartbeat [0] and body temperature.

[0] https://techcrunch.com/2016/12/05/oxehealth-uses-cameras-to-...

Back when they originally came out, I'm sure you'll recall that the operator was basically looking at a high resolution nude image of the person being scanned. The false positives came when people protested and they changed the display to a pixelated cartoon that shows "suspicious" areas as illuminated squares that need to be patted down by a human.

No, those are two totally different technologies. One was x-ray (and terribly dangerous to use in that way) and the current in millimeter wave (which literally cannot make a nude image. The best it can do is the cartoon outline)

This article is possibly where the poster above me read the 54% FP statistic, and its point is not about resolution but that the different wavelength of energy used will show things like sweat as surface, and combined with the automated detection it produces a lot of false positives.

https://www.propublica.org/article/sweating-bullets-body-sca...

In the context of an administrative search at the airport, security guards can already just search whomever. Arguably, the machines are able to clear ~50% of travelers and exempt them from a more intrusive search.

Reply on news.ycombinator.com