Sigh. This article describes a mathematical analysis that is more an artifact of the problem setup than a proof that more evidence decreases confidence. In particular, what's going on is that it assumes that while actual witnesses have a probability of e.g. 50% of convicting, biased witnesses have a probability of 100% of convicting(!), which is an incredibly strong assumption (and in particular, if you changed 100% to anything else, one of the headline results they advertise of unanimity being suspicious and near-unanimity not being suspicious disappears).

Those values of the parameters don't make sense for many real-world scenarios like picking a subject out of a lineup (you'd have to have a really really bad lineup to have witnesses be guaranteed to pick the same wrong person!) or to the output of a panel of judges or jurors.

However, the result is relevant in scenarios where it's very likely that you'd have a 100% likely error, e.g. you're doing an experiment and your apparatus isn't functioning properly or you have a bug in your computer program doing the data analysis. So it's relevant for physicists and computer programmers, but probably not to the criminal justice system.

For those curious, the math is quite simple. They model a world where one of the following happens: * probability p: compromised experiment, in which case fraction y (y=1 in the paper) of witnesses report guilty * probably 1-p: normal experiment, in which case fraction x of witnesses report guilty

Given that, the probability that 100% of N witnesses agree the person is guilty in each case is:

prob(unanimous and compromised) = p * y^N prob(unanimous and not compromised) = (1-p) x^N ~= x^N (since p is assumed small)

So the probability that the experiment is compromised, given a unanimous outcome, scales with their ratio, p y^N / x^N = p * (y/x)^N. A few things you notice: * If y <= x (as you'd hope to be the case in a lineup where the bias is less convincing than having actually seen the criminal!), unanimity is never suspicious. This is the usual world where more evidence should increase your belief in a hypothesis! * If y > x, you have exponential growth and so with sufficiently large N you'll see unanimous results are almost always the result of a compromised scenario. However, unless y/x is large and y is close to 1, the actual probability of unanimity is basically 0 so this will essentially never happen. * If y=1 and x << 1 (the case they analyze), with large N, unanimity is suspicious (but if you change y to 0.95, near-unanimity is, too, so ). This is a case very relevant in software, but less relevant in the criminal justice system.

They picked y=1 and x=0.5, which means roughly you start to see a big effect of this form when N > log_2(1/p), which for their ranges of p from 0.01 to 0.0001 means N ~ 15-30 range.

Unfortunately this is part of a much broader trend of people making bold claims about society based on the results of either a study with a sample a freshman psychology class that achieved 95% confidence (just like ~5% of the thousands of other studies done on psychology classes!) or a mathematical analysis of a confusingly designed problem that doesn't reflect well the broad swath of reality its authors (or the people reporting on it to a wide audience!) are claiming it describes.