Hacker News Reader

> The impossibility is not due to methodological defects in the current state of scientific practice. It's intrinsic to the world. If you cheat only a bit, and only rarely, it might be impossible to detect it statistically.

Not if you take enough samples. Just have to do it enough to reduce the noise.

But to your main point, yeah, in a game of Catan you can just get unlucky. And I think a fair amount of people claim that Catan dice aren't fair in the first place.

Also, did they test the unweighted dice? To understand if there was an inherent and unknown bias in their dice to start with? If they did, I missed that part, and they just assumed their dice were fair to begin with.

I don’t think you are disagreeing with the person you are responding to. They are trying to say the problem isn’t necessarily the ‘scientific process’, it is that if you cheat rarely enough you might not give your opponent enough samples to prove it. Your point that if you had enough samples you could prove it is actually agreeing with them.

Oh, I'm definitely agreeing with the person. I was just being a little snarky at that one statement.

I am also trying to ask a question that is extremely relevant to the author's "study".

"Are the original dice fair?"

That was never asked, it was assumed. It is a pretty hefty assumption too. I think everyone that has played a game of Catan has, at some point, questioned the fairness of the dice. I'm not saying they are unfair, but because of manufacturing methods, it is quite possible to get a pair of dice with an ever so slight bias.

ALWAYS ALWAYS ALWAYS find the bias in the instruments you are using to perform an experiment. THEY ALL HAVE THEM. NEVER NEVER NEVER assume your instruments are accurate without first verifying.

> I think everyone that has played a game of Catan has, at some point, questioned the fairness of the dice.

This is why a lot of people play with a set of 36 cards (one 2, two 3's, three 4's...one 12). There really aren't that many rolls in a game, and there's a lot of variation that may result in a 5 never coming up, which adds more chance to the game than I like. Cards mean that while the order is random, you get each number in the deck eventually.

I've never heard of anyone playing like that before actually. And it seems to completely mess with the statistical nature of the game. Even if you shuffled them each time, shuffling methods are bad and you probably wouldn't get a nice normal distribution. Which fair dice produce.

If you don't shuffle until you run out of cards, you mess with the instantaneous probability of any particular draw, but you hardcode the distribution. You are guaranteed to get the distribution you put in the deck, and I think that's a better measure of fairness than having a constant instantaneous probability.

As the deck becomes smaller, your ability to predict the next draw scales with your skill at counting cards. I think that's a good feature for a random number generator to have, as it's what allows games like poker and blackjack to be about more than simply getting lucky.

Well, I agree that a game needs to have more than luck. But something like blackjack is a horrible example because there's a lot of skill that can be applied even without counting cards. Not only that, but blackjack still has a large amount of luck involved with it even when you are highly skilled.

Personally, I find that the best games are ones that have a good balance of luck and skill. Too much luck and there is no skill, you might as well play a slot machine (some people enjoy games candy land). Too much skill, and winning distributions become too low. A highly skilled player will ALWAYS win, and creates too high of a barrier to entry (connect 4 or dots and squares are an example here). There are (a few) exceptions to the latter like Go, which has so many possible moves that you might as well have an element of randomness, but there is still a steep learning curve.

Catan is one of my favorite games introduction to Euro Games, because the learning curve is low, and there is enough luck that an intelligent novice can win. I don't actually believe the dice are unfair, but the low number of rolls makes each game different. This means the skilled player needs to be highly adaptive to the changing environment.

Dice create a nice normal distribution that are independent. While over a large number of games, 6 and 8 are great choices, there will be games where you just don't do well (they are rare). By not shuffling the cards, you are creating a flat distribution and really removing the vast majority of luck in the game (you still have luck in the order of the cards, order of placement, and order of turn). You now have a dependent probability function, and I think you could make great arguments that you remove all the things that (I believe, and laid out above) make the game great. I think you could also make arguments that the setup is the most important part of the game (when your cards are dependent events). But it is a game, and these are just opinions.

It absolutely changes the game, since you can now plan that an unlikely roll will definitely come up at predictable times.

The author was trying to analyse the effect of loaded dice on a game of Catan, not test whether the water-soaking method produced loaded dice. It doesn't really matter where the loaded dice came from.

All dice are slightly loaded.

Statistics is fine, people who are bad at statistics are the problem. When it comes down to it, many "scientists" are rather poorly educated.

Indeed, statistics is fine, but I think the problem is larger than scientists being poorly educated. It looks to me like an awful lot of scientists aren't acting in good faith. I don't think that their conduct necessarily rises to the level of dishonesty, more like they aren't taking obvious steps to mitigate the potential for erroneous results.

> aren't taking obvious steps

Because none of their peers think those steps are obvious, and their peers determine their funding, status, etc.

Yeah, the idea that being able to adjust the priors for a Bayesian analysis based on suspicious behavior, and then saying that proves cheating, is certainly not the case. The cheater is not going to agree with your assessment of their ‘suspicious’ behavior. You might as well just skip the analysis and say you proved they cheated because of their behavior.

>The impossibility is not due to methodological defects in the current state of scientific practice. It's intrinsic to the world. If you cheat only a bit, and only rarely, it might be impossible to detect it statistically.

I don't understand what you're saying or maybe I just want to go further. But I would say it's an ill-posed question whether someone's cheating in that sense. They could win a one in a million chance repeatedly over and over on their first try and it wouldn't be proof that they're cheating. It's not even necessarily probable. The probability that they're cheating depends not just on how good their results seem to be but also other things, primarily: on your priors about how likely it is that they would cheat.

If a known cheater gets a string of mild good luck it is more indicative of cheating than if a known honest person does. Bayesian reasoning has to be used for this right?

Well said. It doesn't reveal problems with current methodology, it reveals problems with its application, and understanding. Larger sample sizes would fix quite a bit. Studies, even "pilot" or "priming" studies, are not reliable at a .05 significance. Especially when most of the false positives will get published, but nearly none of the false negatives will see the light of day.

More practically, all dice are slightly loaded.

I think there are other statistical methods that would take account of sample size in different ways -- perhaps saying in some cases "you don't have enough data to determine validity yet". Among other things that different tests would reveal. p-value is not the only way to analyze your results statistically.

p-values say a certain thing, and are not appropriate to be used as the single universal statistical test for validity, and may likely not even be the best 'default' choice.

I think the message here is indeed that statistical verification/validation is _hard_, and requires actual statistical expertise to know what the tests you are applying mean (and what the assumed pre-requisites of the test are) and apply the right ones. The current practice of "p-value and only p-value, always" is flawed.

(But it's also because the influences of current scientific/academic practice are to motivate people to find the _quickest and cheapest way to get published_, rather than to give yourself additional barriers to publication in pursuit of truth!)

There has been a lot of writing about this in more formal and less cutesy settlers-example ways, this is not an original observation to this blog post. It's in fact been a growing concern, including leading to a recent statement of caution on over- and mis-use of p-values by the American Statistical Association.

http://www.nature.com/news/statisticians-issue-warning-over-...

https://www.sciencedirect.com/science/article/pii/S106345841...

http://blogs.plos.org/publichealth/2015/06/24/p-values/

http://www.ejves.com/article/S1078-5884(15)00585-7/pdf

http://sphweb.bumc.bu.edu/otlt/mph-modules/ep/ep713_randomer...

I like this as a demonstration of one way p-values can be insufficient or lead you wrong, as people have been increasingly discussing.

It would be interesting to apply some of the other suggested statistical tests (confidence intervals?) to the settlers experiment results.

> perhaps saying in some cases "you don't have enough data to determine validity yet

Contrary to popular belief, that's actually exactly what a p-value is. A p-value determines whether your sample size is large enough to reject the null hypothesis, given the observed variance in the sample data.

Introductory statistics classes don't really explain this properly, because it would require knowledge of continuity theory that most don't have at that stage, but that's really what a p-value is already telling you.

I admit I'm not a statistician (or someone that regularly does quantitative research) and don't really understand what a p-value says (without refreshing myself on it for reading for 30 minutes each time heh). So thanks for correction.

But I think many of the researchers applying it don't either, heh.

The real point is just that there are other statistical tests that might be appropriate here, and perhaps would be more reliable at catching this. And that there's actually widespread agreement among experts that p-value testing is widely wrongly used.

> large enough to reject the null hypothesis

Or at any rate, that there's a calculated max 5% chance of the null hypothesis being true, if the assumptions/pre-requisites of the p-value test were actually met. Which is still actually kinda large, 5%, depending. And as I try to think about and confuse myself, may actually be entirely the wrong test as applied here, not sure.

> But I think many of the researchers applying it don't either, heh.

As a statistician myself, I'd agree with that statement.

I've actually said previously - and only half-joking - that p-values should be banned from research journals. The problem with p-values is that they don't mean what people generally assume they do, and because they look so close to what people want them to mean ("the probability that my conclusions are wrong, given my assumptions and data"), it's very easy to project spurious meaning onto them.

In reality, p-values actually provide very little information, and the information they do provide is generally not of relevance. But they're so commonly-used that it's very hard to convince people to use more sophisticated techniques for reporting and modeling information.

I guess it won't change until the 'peer-review' process for every article includes a statistician, and papers actually get rejected for improper or insufficient use of statistics.

I think academia these days forces researchers to really care about little except getting grants and getting published (with the former effected by the latter) -- caring about using statistics properly (let alone the actual validity or usefulness of their findings) will hurt rather than help their careers unless it effects one of those two things positively.