Using loaded dice to cheat at Settlers of Catan, and p-values to avoid suspicion

p-values say a certain thing, and are not appropriate to be used as the single universal statistical test for validity, and may likely not even be the best 'default' choice.

I think the message here is indeed that statistical verification/validation is _hard_, and requires actual statistical expertise to know what the tests you are applying mean (and what the assumed pre-requisites of the test are) and apply the right ones. The current practice of "p-value and only p-value, always" is flawed.

(But it's also because the influences of current scientific/academic practice are to motivate people to find the _quickest and cheapest way to get published_, rather than to give yourself additional barriers to publication in pursuit of truth!)

There has been a lot of writing about this in more formal and less cutesy settlers-example ways, this is not an original observation to this blog post. It's in fact been a growing concern, including leading to a recent statement of caution on over- and mis-use of p-values by the American Statistical Association.

http://www.nature.com/news/statisticians-issue-warning-over-...

https://www.sciencedirect.com/science/article/pii/S106345841...

http://blogs.plos.org/publichealth/2015/06/24/p-values/

http://www.ejves.com/article/S1078-5884(15)00585-7/pdf

http://sphweb.bumc.bu.edu/otlt/mph-modules/ep/ep713_randomer...

I like this as a demonstration of one way p-values can be insufficient or lead you wrong, as people have been increasingly discussing.

It would be interesting to apply some of the other suggested statistical tests (confidence intervals?) to the settlers experiment results.

Not if you take enough samples. Just have to do it enough to reduce the noise.

But to your main point, yeah, in a game of Catan you can just get unlucky. And I think a fair amount of people claim that Catan dice aren't fair in the first place.

Also, did they test the unweighted dice? To understand if there was an inherent and unknown bias in their dice to start with? If they did, I missed that part, and they just assumed their dice were fair to begin with.

I don't understand what you're saying or maybe I just want to go further. But I would say it's an ill-posed question whether someone's cheating in that sense. They could win a one in a million chance repeatedly over and over on their first try and it wouldn't be proof that they're cheating. It's not even necessarily probable. The probability that they're cheating depends not just on how good their results seem to be but also other things, primarily: on your priors about how likely it is that they would cheat.

If a known cheater gets a string of mild good luck it is more indicative of cheating than if a known honest person does. Bayesian reasoning has to be used for this right?

Hmm the dice are behaving funny, but that's fine, everything is just fine...

https://izbicki.me/blog/turning-an-ak-47-into-a-serving-ladl...

Large changes in bet-size are an obvious indicator, so card counters must work in teams. One is the counter, the other gambler is the whale who bets large when the count is favorable.

Unfortunately, the casino has a simple defense: table limits. The whale must sit at the high-roller table, the counter must sit elsewhere. Count cards as much as you like, but you'll never make money at a modern casino ("you" being the aggregate of all card counters, not an individual).

I have a hard time believing that after 100 games someone would say "you know what, it appears to me that sixes have been rolled slightly more over the previous 100 games than I would expect" and an even harder time believing the next logical would be "you must have altered the dice!"

https://en.wikipedia.org/wiki/Pearson%27s_chi-squared_test

GM's don't have to show their dice rolls and can overrule the outcome if it fits their storyline better, so the rolling is mostly ceremonial anyway.

And if you slice the data from a single experiment in 40 independent ways, your chances to get something with purely random significance p<0.05 are better than 50% for a single study…

That would only be the case if scientists were robots who immediately published anything with a p-value up to 0.05. They're not, though. If they get clearly nonsensical results, they will obviously re-evaluate it. In other words, the p-value doesn't incorporate the fact that the experiment passed sanity checks in your own head (and the reviewers') before it was published. (And yes, there are bad actors in every field who game the system, but my point still stands.)

How so? I don't see any connection between significance level and % of false scientific results at all.

If you assume the "null hypothesis" is always true, then 5% of the results should falsely say otherwise. Of course, this is if all the assumptions behind the math hold, no p-hacking, etc.

However, that is like saying it is extremely rare for there to be a correlation between any two phenomena. We don't live in that universe. In our universe, correlations are extremely common:

>"These armchair considerations are borne out by the finding that in psychological and sociological investigations involving very large numbers of subjects, it is regularly found that almost all correlations or differences between means are statistically significant. See, for example, the papers by Bakan [1] and Nunnally [8]. Data currently being analyzed by Dr. David Lykken and myself, derived from a huge sample of over 55,000 Minnesota high school seniors, reveal statistically significant relationships in 91% of pairwise associations among a congeries of 45 miscellaneous variables such as sex, birth order, religious preference, number of siblings, vocational choice, club membership, college choice, mother's education, dancing, interest in woodworking, liking for school, and the like. The 9% of non-significant associations are heavily concentrated among a small minority of variables having dubious reliability, or involving arbitrary groupings of non-homogeneous or nonmonotonic sub-categories. The majority of variables exhibited significant relationships with all but three of the others, often at a very high confidence level"

-Theory testing in psychology and physics: A methodological paradox. http://www.fisme.science.uu.nl/staff/christianb/downloads/me...

The 5% figure means that, when there is no signal to detect, we have a 5% chance of falsely claiming there is one. It does not say anything about the case when there is a signal and we do not detect it, which is known as the type II error rate.

With reasonable assumptions about sample size and the fraction of times there really is a signal, you can find that the majority of published results are false:

http://journals.plos.org/plosmedicine/article?id=10.1371/jou...

I have written an intuitive explanation of this, and a bunch more, in my book: https://www.statisticsdonewrong.com/p-value.html

Originally, I was going to point out other null hypotheses that we could try to reject. This has the advantage that some of them (like the one you propose) model the physics. And it also has the disadvantage that if we propose too many hypotheses to test, then we will be more likely to get false positives. But the article was already too long, and this is a HUGE can of worms to open.

I feel like a game is fair if all players know that a die is biased, but no player has an advantage in predicting the outcome.

To achieve that: for each die, for each roll have the players agree on a random permutation of the observed faces to actual faces for that roll. As a result even if all players know that 6 will land with probability 1 in the original throw, after permuting each side has probability 1/6.

The agreement algorithm: have the parties each propose a permutation, and compose the permutations together in order to get a final "unbiased" permutation. I think if there is at least one honest party, this will work.

Another approach is to let everyone sample the dies before playing until they are satisfied, and let everyone adjust their strategy accordingly.

You can use this technique to unbias a 4-die or 8-die by throwing a coin multiple times (apply the above technique for each digit in base 2).

To produce a fair 6-die from a biased coin, the following should work.

- Using the Von-Neumann technique, creates 3 independent unbiased coin X, Y, Z. Then X= 1 + X + 2Y + 2Z is uniform in {1,...,8}, we basically randomize each digit in base 2.

- Finally, throw the unbiased 8-die until you obtain a number different than 7,8. The result is unbiased in {1,...6}.

    import numpy as np
    
    def biased_coin():
        return np.random.choice([0,1], p=[0.1,0.9])

    def unbiased_coin():
        a = biased_coin()
        b = biased_coin()
        if a != b:
            return a
        else:
            return unbiased_coin()

    def unbiased_eight():
        return 1 + unbiased_coin() + 2*unbiased_coin() +4*unbiased_coin()

    def unbiased_six():
        d = unbiased_eight()
        if d < 7:
            return d
        else:
            return unbiased_six()

    np.bincount([unbiased_six() for i in range(6000)])

If you're playing Catan by email, this approach can be extended with bit commitment. Each player rolls a die and sends a message to the other players with the number they rolled signed with their own digital signature but also encrypted with a key randomly chosen for that round of die rolls.

None of the other players can know what the other players rolled without the key. Once everyone has sent all the other players their rolls (i.e. they've "committed" to a particular die roll but the other players don't know what it is yet), they exchange keys to reveal the numbers they rolled.

The method is an extension of von Neumann's trick for un-biasing a biased coin in which you flip it multiple times and take advantage of symmetry (specifically flip it twice and return heads if you got heads then tails, return tails if you got tails then heads, or repeat the procedure if you obtained two heads or two tails).

You can then go from this bit-stream to a number in {1,...,6} using the trick jknz mentions below (take 3 successive bits, and interpret them as a binary number between 0 and 7, add one to get a number between 1 and 8, and start again if the result is 7 or 8).

[The method in the paper is apparently asymptopically optimal (as the number of faces on the dice goes to infinity). However, for small numbers of faces I think it is slightly worse than the method jknz described. Consider the 4-faced dice: the Peres method converts a pair of dice rolls into 1 or zero bits, but jknz's method would sometimes produces two bits (the rolls 1,4/2,3/3,2/4,0 are binary 00,11/01,10/10,01/11,00 resulting in sampled bits 0,0/0,1/1,0/11). Working out all the cases shows this makes the method slighly more efficient.]

[1]: http://dx.doi.org/10.1109/TIT.2014.2381223

They also have Carcasonne, which they call Tolouse.

Your opponents can't prove cheating based on the game alone.

If you're going to cheat, you're better off sleeving your most critical resource cards, to protect them from 7-rolls, soldiers, and monopoly cards taking them from your hand. You unsleeve and swap when you get a favorable resource roll, and hope nobody is OP enough to keep a running total of all opponent resources in their head.