Comment by ajross | Hacker News Reader

Comment by ajross | Hacker News Reader

Parent Full thread

> Also, suspicion was by only a 1:100 coincidence probability standard of evidence, and by some imperfect metric.

...only? In a sample size of 96, what's the value in analysis beyond that level?

No, this seems pretty solid to me. The math to do this is routine freshman statistics stuff, something every practicing scientist knows. The assumptions just require that wrong answers be roughly evenly distributed across the choices (e.g. you could construct a test where everyone who was wrong would be led to choice C and rarely B or D, but that's a little pathological; and regardless it's something that would be evident in the data set and seems not to have been).

I mean, standards vary but I'll bet in most jurisdictions 100:1 odds count as "beyond a reasonable doubt" for jury instructions in criminal trials. At the very least you'd bring the trial, which is what happened here. If I know there's only a 1% chance I'm wrong, it's absolutely valid for me to accuse you of cheating.

Actually, I can't seem to get the maths to work. Isn't it just the Binomial distribution? Each question has 5 options so the probability of 'success' is 1/5 and so to get 19 questions 'specifically wrong' out of the 45 planted questions by chance is just (1/5)*19 * (1-1/5)*26 * 45!/(19!26!) = 1 in 2588 but in the article it is 1:100. What am I doing wrong ?

Here's my best guess: 19 is the cutoff point for a binomial test [1] where the probability of at least that many answers to match those in the honeypot test goes below 0.01. But this holds only if you assume p=0.25.

Why would you use 0.25 instead of 0.2? I guess it would make sense if you only looked at wrong answers - that is, you wouldn't be asking "what's the probability of your answers matching those on the fake test" but rather "what's the probability of your wrong answers being wrong because you used the fake test". Since you are only looking at wrong answers, your probability is 1 in 4 instead of 1 in 5.

[1] At least, according to this calculator: https://www.socscistatistics.com/tests/binomial/default2.asp...

"1% false positive rate => 99% confidence about a positive signal" is a really intuitive step to take, but you should be careful because it turns out to be horribly wrong in general: https://en.wikipedia.org/wiki/Base_rate_fallacy

(Perhaps not in this case but in general this can get you into trouble)

Though personally I'd say that it's quite likely that the p=0.01 threshold falsely flagged a non-cheater, which to me says it is too lax.

> I'll bet in most jurisdictions 100:1 odds count as "beyond a reasonable doubt" for jury instructions in criminal trials.

IANAL, but I sure hope that 100:1 chance, standing alone, doesn't meet the bar for "beyond a reasonable doubt".

To me, a 1% chance of an accusation being false is definitionally "reasonable doubt".

...I have no idea where I heard this, but for some reason I was under the impression "beyond a reasonable doubt" meant ~95% confident. (Whereas "preponderance of evidence" meant >50% confident.)

Per Wikipedia, the original framing is Blackstone's ratio from the enlightenment, where he used 10:1. Real courts don't put numbers on this, of course, but that's the thinking that produced the philosophy. Basically: no, given the level of innaccuracy we already know to be present, I think it's very clear that a sincerely-believed 99% confidence will send someone to jail pretty much anywhere in the US.

https://en.wikipedia.org/wiki/Reasonable_doubt

https://en.wikipedia.org/wiki/Blackstone%27s_ratio

The quoted sentence I objected to was the sloppy writing, conflating of "suspected" with "caught". I think "caught" would be read by most readers as implying guilt.

Suspected at 1:100, certainly. Quietly confronted and questioned at 1:100, sure. Convicted at 1:100, absolutely not.

I'm not sure this is the best analogy, but imagine that, for every 100 people in society, 1 of them is wrongly convicted, at random. Further imagine that this lottery happens repeatedly, so that even more than 1 in 100 end up getting hit. That sounds like a miserable society in which to live.

Headline: "80 million people worldwide caught cheating" just in the first round.

Reply on news.ycombinator.com