TSA to Commission Independent Study of X-Ray Body Scanners | Hacker News Reader

TSA to Commission Independent Study of X-Ray Body Scanners

propublica.org

58 pointsceejayoz57 comments

Open articleView on HN

Assuming it's a real study and they actually gather new data and run some experiments, what are the possible results?

1) The machines are unsafe and should be removed immediately

2) The machines are unsafe for people that pass through them at some given frequency (like 5x a year or something)

3) The machines are completely safe

4) ?

Results 1 and 2 would mean that many air travelers would feel entitled to some sort of compensation.

There's going to be a lot of external pressure to find result #3, so I'm glad it's an 'independent' study.

In all likelihood, the conclusion will be #3. If government agencies feel compelled to do such studies beforehand from now on, then change has occurred in the right direction.

If it's (3) saying in _zero_ harm then it's almost certantly a lie (http://en.wikipedia.org/wiki/Linear_no-threshold_model), if it's a (3) saying the harm is very low and so it wins on cost benefit — well, that is possible but it seems really unlikely considering how insanely low the base rate of terrorism is, much less the subset of attacks which the scanners actually prevent (vs just shuffling around the attack mechanisms).

it seems really unlikely considering how insanely low the base rate of terrorism is, much less the subset of attacks which the scanners actually prevent

This is an argument for not screening passengers, or at least doing it very differently from the way we do now. It's not an argument for using one type of scanner rather than another, given that you are going to screen with scanners. That's the only question at issue in the OP: whether TSA should use X-ray scanners or another type.

The cost/benefit is likely to be different between the X-ray backs after and millimeter wave scanners.

Different, sure. But different doesn't necessarily mean better.

Explain how risk/reward could be different without one being better than the other. Is one going to come out as an imaginary number or something?

Actually, now that I think about it more, you're probably going to be dividing by zero to calculate that risk over reward so testing equality will be a challenge...

Because risk/reward requires tradeoffs; it isn't a single linear scale. Different tradeoffs can be judged differently based on who is doing the judging and what criteria they are using. There isn't a single "right answer".

For anyone's subjective answer, one is going to be better than the other, unless they are freakishly identical. Whoever provides the most rational defense of their analysis is the person we listen to.

We don't need to play fast and loose with the calculation though. Real data would be trivial to accumulate, were it permitted.

If more people are likely to die per year from cancer caused by the scanners than are likely to die from terrorists, I think that's a convincing argument that using the scanners screen passengers is worse than not screening them at all.

using the scanners screen passengers is worse than not screening them at all.

Sure, that's true (given your assumptions) if the scanners pose any risk that's worse than just walking a distance equivalent to the space taken up by the checkpoint. But as I said, that's not the question under discussion; the question is whether there is any significant difference between one type of scanner and another, given that you're going to scan passengers.

It's actually plausible that the scanners - if working as designed - might cause net health benefit ( http://en.wikipedia.org/wiki/Radiation_hormesis http://en.wikipedia.org/wiki/Hormesis ).

The tricky thing is that we have no way of knowing if the machines are operating as designed and the powers that be have a looooong history of outright lying to passengers. I still remember when they told people sending your camera through the X-ray wasn't bad for the film - this was a lie, but it made security's job easier to tell that lie, so they told it. Knowing that the TSA is a bunch of incompetent and untrustworthy nitwits, I don't trust them to operate dangerous equipment safely. Since a machine that's not being maintained correctly or isn't being used correctly could be really dangerous, I generally choose to opt out.

There are tons of different possible results, depending on what they measure.

Maybe the machines are dangerous for the people who work around them every day. Why aren't these people wearing dosimeters?

Maybe the machines have an effect that only appears in a very small percentage of individuals.

Maybe the danger only comes after the machine drifts out of calibration, while in a humid room, operated by poorly trained persons who misinterpret a control panel, when the person scanned has a given gene, that is being expressed because they have a viral infection.

Conclusion 3(that you listed) can only be considered crap. To arrived at it is to claim to have a perfect understanding of several very complex systems. No matter how many studies you perform that fail to find a danger, that is not evidence that no danger exists.

"The absence of evidence is not evidence of absence."

The most you can ever say is "We found no evidence of dangers."

My issue is with what threshold of safety should be considered. If for example, the X-ray dose causes one fatal cancer in a million, it would be a positive risk-benefit for many medical procedures, but for the rate of actual terrorism attempts vs air travelers getting that dose millions of times over, the risk benefit seems pretty negative. This is especially true if you're a frequent air traveller and could be dosed two or more times per trip. That really means that airport X-ray machines should be held to a much higher safety standard then medical x-rays.

My issue is with what threshold of safety should be considered.

That's a good question. The numbers TSA gives, which are quoted in the article, are that a single exposure to the X-ray scanner gives you the same radiation dose that you get from two minutes of airline flight. So if we think the risk of the scanner is unacceptable, we should also consider it unacceptable for a flight to be lengthened by two minutes. Flight times routinely vary by more than that due to re-routings for things like weather, air traffic control, etc.

Nothing is 100% safe. If we are going to try to consider thresholds of safety, we have to consider all possible risks, not just the ones the media is currently harping on. What we find when we do that is that we routinely accept risks far greater than the ones the media is currently harping on. But those risks don't sell papers or keep people watching.

The minutes of flight are there because they're a fundamental part of travel and are unavoidable if you choose to travel by air. An x-ray machine is there only for the purpose of additional security. If the risk undergone for the machine outweighs the security benefit then the machine can and should be removed. In my mind, there is no equivalence between the unavoidable risk and the completely avoidable risk.

A lot of flight time is unnecessary. The technology to more efficiently route airplanes exists, we just have not yet allocated the money to do it. If it turns out that the X-rays really are no worse than 2 extra minutes of flights, I'd rather put the money towards improving the air traffic control infrastructure. Then we save time, reduce radiation exposure, and likely reduce the risk of an airplane crash. Seems like a better use of money to me.

Great point -- this is exactly the sort of thing I was talking about.

Airlines are already optimizing for flight times - time in the air is fuel burn which is a major cost input for the business.

If your purpose is to save lives, and have some amount of money you're intent on spending to do so, there are many better opportunities to get better bang for your buck than either airport x-ray machines, or shortening flight times.

What really bothers me is that I think that politicians and TSA bureaucrats have placed an outsized symbolic value on air security via X-ray machine, and setting in place a policy which causes real harm in sicknesses that are essentially unlinkable to the machines. Without a detailed and critical look at the effects, there will be no obvious red flags that people are literally dying for symbolism.

Airlines are already optimizing for flight times - time in the air is fuel burn which is a major cost input for the business.

Airlines certainly try to do this, but they don't control either the weather or the air traffic control system. That said, I don't disagree that further optimization for flight times is likely to have a low payoff compared to lots of other things. My only point is that any flight time optimization that's worth more than two minutes per flight, on average, must, of necessity, have a higher payoff than changing the type of scanners at the checkpoint.

politicians and TSA bureaucrats have placed an outsized symbolic value on air security via X-ray machine

This argument applies to any type of scanner, not just an X-ray one. It applies equally well to the millimeter wave scanners that the article mentions as the chief alternative to X-ray scanners. And it in no way affects the point that the additional risk due to the X-ray scanner, given that you have already accepted the risk of a flight, is negligible.

> The technology to more efficiently route airplanes exists, we just have not yet allocated the money to do it.

This is actually well on its way to being implemented.

http://en.wikipedia.org/wiki/Automatic_dependent_surveillanc...

The minutes of flight are there because they're a fundamental part of travel and are unavoidable if you choose to travel by air.

You're missing my point. My point is that if you think the radiation risk posed by the scanners is significant, you should not be flying in the first place, because the radiation risk posed by the flight is much more significant. If you've already decided to accept the radiation risk posed by the flight, the additional risk posed by the scanner is negligible; you would be better off spending the time and effort you are spending protesting about the scanners on something that would have more impact. (For example, see the post just downthread by krschultz.)

An x-ray machine is there only for the purpose of additional security. If the risk undergone for the machine outweighs the security benefit then the machine can and should be removed.

But that's not the question under discussion. The question is which type of scanner TSA should use, not whether TSA should scan, period.

The risk probability is small for an individual trip. That doesn't mean we get to avoid looking at the collective risk numbers for scanners. In 2011, there were 640 million domestic air passengers in the US. If as a policy we're going to perform 640 million scans, what level of harm should be considered safe? If we take for example, win-the-lottery type odds of one-in-a-million, that's pretty small individually. Now perform the independent random draw 640 million times per year - that would be 640 cases of premature deaths per year give or take a little variation. If no scans are performed that outcome of additional deaths would be zero. Small numbers are not negligible when you repeat the trials millions of times.

The risk probability is small for an individual trip. That doesn't mean we get to avoid looking at the collective risk numbers for scanners.

And exactly the same logic applies to flying in general. If we don't consider the harm due to the scanners to be safe, then we shouldn't consider the harm due to flying itself to be safe, since the latter is several hundred times as large (at least) as the former. Both are being aggregated over exactly the same population of people.

And before you say that "I choose to fly, but I don't choose to be scanned", you don't choose what the attendant risks of flying are either. You just have to decide whether or not to fly based on the risks as they are. Basically, with the scanners, when you choose to fly, you are choosing to fly for a couple more minutes than you would in the absence of the scanners. Do you routinely make flight choices based on differences of minutes in flight time? I thought not.

One concern that has been raised is that while the radiation is weak, it does not go through the body but is localized in the skin, and that the machines were not properly certified as medical devices, so there was no way to know whether it leaked radiation, would malfunction in an unsafe manner.

I think radiation concerns may be more relevant for people who have already had many X-rays due to disease, or workers who go through or have to be around these machines all day.

My primary objection is that the cost of the machines isn't justified in the security improvement and potential privacy threats, and it's still easy to fool it as long as the X-ray isn't through-body. Getting through-body scanners certified and proved to be safe is going to be even more difficult, no matter how few minutes of flight you can compare it with.

Yes. However, nobody's considering the rate of actual terrorism attempts. If they were, we'd remove every dollar of the TSA's budget and spend it on highway safety.

Or, if we specifically wanted to spend money guarding potential terrorism targets, we'd spread our security dollars over airports, schools, stadiums, water and power plants, shopping malls, bridges, office buildings, parking decks, subways, concert halls...

... see how impossible it is to actually protect everything? Luckily, it's not necessary in the first place.

Those are the results of the study, but what I'm more curious about is what will come out of the findings. That is, if the machines are found to be unsafe to some level, will the TSA actually remove them from use?

4) Study finds the scanners are unsafe, contradicting what the companies selling the scanners claim. TSA sues companies providing the scanners and makes them the scape-goat

Perhaps it's wishful thinking, but one can hope...

Too bad they won't do a study of how useless the entire TSA is. It wouldn't be hard: just randomly choose a group of airports that are allowed to go back to the previous model of private inspectors hired by the airlines. For every year that goes by with the experimental group of airports having better financial results, better security, smoother operation, and fewer passenger hardships, a few more airports will win the lottery and be allowed into the experimental group. Eventually the TSA will be studied out of existence.

Well the entire purpose of the TSA is security. And how do you evaluate security? The number of successful attacks on airports?

Hopefully both the test set and TSA set would have a result of zero. And even if they didn't, you can't say for sure that it was because of TSA presence. So the test would be inconclusive, and we'd end up keeping them around "just in case". After all, we haven't had another attack like September 11th, so the TSA must be working, right?

If you define security as "how many commercial airplanes have been flown into tall buildings", you're probably right. However, I'd also like to include such measures as "how many items have been stolen by TSA agents from luggage", "how many times does the TSA violate its stated policies", "how many bins of 'potential explosive materials' are stockpiled at security checkpoints", "how many times do red-team tests result in missed dangerous items", etc.

Really I think you could use any measure of security, even one picked out by TSA personnel, and the TSA as currently constituted would do worse than a private contractor reporting to the airlines. (Even if I'm wrong, wouldn't we want to know?) The important thing is that security must be balanced against the other things I mentioned: cost of the security program, indirect costs borne by airports and airlines, and passenger experience. From a societal perspective, we also have to weigh the costs associated with encouraging travelers to drive rather than fly.

how many items have been stolen by TSA agents from luggage

As far as I know, the amount of theft due to TSA is negligible; virtually all of it is due to airline employees.

I suggest googling for "tsa theft"; if they're firing hundreds of people a year every year it isn't a negligible problem. But even if more is stolen by airline employees than by TSA agents, that isn't a defense of the TSA. We couldn't fly without the airlines; we flew for decades without the TSA.

if they're firing hundreds of people a year every year it isn't a negligible problem

From what I can see from a Google search, "hundreds" is the total number of TSA people who have ever been fired for theft, not the number per year.

Given the the low end of "hundreds" and the time the TSA has been in existence, that is still not negligible.

Particularly if they suck at catching/firing people as much as they do at any other sort of screening.

The important thing is that security must be balanced against the other things I mentioned

Which is exactly what wouldn't happen. The security theatre is persuasive to a great many people.

Cost benefit analysis. The cost is huge, the benefit is negligible.

Spend the money on something else: (hello healthcare!) High cost, huge benefit.

The cost is huge, the benefit is negligible.

How are you evaluating benefit? We haven't had a major terror attack using planes since the TSA was brought into airports.

(yes, I'm playing devil's advocate here, but only because that's exactly what would happen)

Tiger repellent rocks, etc.

Seriously though, consider the false negative rate they have (by some accounts, as high as 70%), and the cost associated with actually being caught (they throw your knife into a bin, then wave you through onto the airplane). It just isn't plausible that the TSA is making terrorists to afraid to try.

Either the terrorists just don't care about hijacking the planes anymore (fairly plausible, planting IEDs is probably more fun), or something else is stopping them. Perhaps the thought of being dog-piled the second the passengers figure out what's up...

> And how do you evaluate security?

It's easy: penetration testing.

With penetration testing you can determine the rate at which a baddy can expect to get by the TSA undetected. From what little of this testing has been done for the TSA, it seems the TSA performs abysmally.

Combine that with the number of false positives (thousands, if not dozens of thousands, a day), and the astronomically small baseline rate of terrorists in the population, and you get the chance that anyone the TSA catches is actually a terrorist. That (absurdly small) probability will tell you a lot about just how worthwhile the TSA is.

<tinfoil_hat>

The plan is for the scanners to fail the tests

TSA apologizes

Congress indemnifies scanner manufacturers

TSA buys new scanners, which is the overall goal

Scanner manufacturers aka campaign contributors profit

Congressmen get reelected

</tinfoil_hat>

Pretty much.. You knew there had to be a version 2.0 that we had to buy at some point.

To be honest, the whole process becomes invalidated when you can now pay via a program called TSA PreCheck (http://www.tsa.gov/tsa-pre%E2%9C%93%E2%84%A2) to skip the lines entirely.

Theater at its finest.

The company that makes these scanners will just take some senators on a few more all-expenses-paid trips to Hawaii for some "conferences" and things won't change at all.

Hawaii is too mainstream, more like the Mariana Islands:

http://www.dailykos.com/story/2007/10/23/401675/-Meet-the-Ta...

My biggest concern is regarding the internal software and dosimetry controls. Could there be a fault condition that would result in unintended levels of ionizing radiation? It wouldn't be the first such accident..

http://bjr.birjournals.org/content/78/934/913.full

http://www-pub.iaea.org/MTCD/publications/PDF/Pub1207_web.pd...

...And the Therac-25 / airport scanner comparison comes full circle: http://news.ycombinator.com/item?id=4919666 =)

> But it is unclear if the academy will conduct its own tests of the scanners or merely review previous studies.

The most important fact that could possibly be presented, and it's not presented.

Signal to noise ratio is basically nonexistent.

Sadly, there's only one way to test these machines right. And that's in the field, as they're being used, without any knowledge that they're being tested.

I'm sure that's not what they're planning to do.

If that is the only way, then the conclusion is obvious.

Can you elaborate?

I don't think I'll trust the results of this. It sounds too much like they found an "independent" group to study it, and then pressure them into giving them the results they want.

I would've rather seen an independent group take the initiative themselves and ask TSA to let them do the testing, at least.

One can only imagine the level class action lawsuit if they were to find them detrimental to peoples health.

The National Acadamy of the Sciences is not just some random "independent" group. Of course, the article states that it's still unclear as to whether they will take more data, or simply review existing studies. Likely, they will do a review first to decide if further study is necessary (e.g., if the methodology used in prior studies was, in fact, unsound). In any case, having unaffiliated scientists consider the issue is a step in the right direction.

It's a bit like if I have commissioned a study on whether or not my new startup is awesome. It could be good if I wanted to find out the truth, but it's worthless if I am ordering one of those for external consumption. He who pays the money orders the music.

My biggest problem with the scanners is that they are a waste of taxpayer money, so I'm not particularly thrilled by spending more taxpayer money to determine exactly how much a waste of taxpayer money they are.

Sunk costs are just that... sunk. Spending X money now to (hopefully) prevent spending X + Y money in the future is ideal.

I doubt the commission will study the 4th amendment constitutionality of the scanners.

Reply on news.ycombinator.com