Hacker News Reader

Comment by r12343a_19 | Hacker News Reader

> we recruited 501 American participants from Amazon Mechanical Turk

The Mechanical Turk folks are the new sociology students. This study has 100% chance of not being reproducible or statistically representative.

> 63.3% of the participants reported having flipped the winning outcome [...] in a previous study using a similar coin-toss game with an online (MTurk) sample (Schindler and Pfattheicher, 2017), 76.7% of the participants reported having flipped the winning coin-side

So, just world people are 13% more honest than the general population? How can you name the title then "General Belief in a Just World Is Positively Associated with Dishonest Behavior" when clearly, everybody is dishonest and these bunch were actually... less so?

> Given that general BJW was found to be linked to antisocial tendencies, we expected stronger general BJW to be linked to more dishonesty.

I postulate academia strongly correlates with dishonesty.

View on HN

No, 63.3% of all participants got a favourable outcome, so 13% cheated (suggesting 26% would cheat if necessary, as half got a favourable outcome without cheating).

The participants all had varying degrees of belief in a just world which were measured with a six-item test. The results of that test correlated significantly with the chance of a favourable self-reported coin toss.

It is meaningless to draw inferences from comparisons to another study, as they were selected differently so there's no control, and it isn't necessary as this study contains a random and varied sample of participants.

Yes, there is a question over the representativeness of Mechanical Turk workers. But even though they're surely different on average from the average human, this study is controlled by comparing them to each other rather than to some pre-determined statistics, so the sampling bias should be largely cancelled out, barring second-order interactions.

> Yes, there is a question over the representativeness of Mechanical Turk workers.

There are more fundamental issues. Having run several studies either with MTurk or panel providers who use MTurk to source some/most of their respondents, I have trouble trusting any study with an MTurk sample that doesn't explicitly show 1. how they verified respondent location and demographics and 2. how they controlled for bots and mindless click-throughs.

Even though they used a convenience sample, issues like reading comprehension (which you can get from non-native English speakers VPN'ing through a US-based IP) and participants trying to get through the study as quickly as possible - or automating their responses altogether - absolutely matter.

I post fairly often here about how terrible different studies are. And many are pretty bad!

This... is a horrifically bad take on the study.

> So, just world people are 13% more honest than the general population? How can you name the title then "General Belief in a Just World Is Positively Associated with Dishonest Behavior" when clearly, everybody is dishonest and these bunch were actually... less so?

What!?

How can you arbitrarily compare another study that used a different experimental design to this one? Just pick out two percentages that are totally unrelated, subtract them, and then complain about something?

This is like saying, 30% of bananas in my house are bad. 42% of cars on my street are red. Why do I have this huge 12% gap between the two!? What nonsense.

> I postulate academia strongly correlates with dishonesty.

I postulate that reading comprehension is important! And that maybe, if you don't know what you're talking about, you shouldn't dump on people's work?

That being said. There are issues with this study. In addition to MTurk being noisy, the primary problem is that the effect size is very small! The authors deftly hide this in the standard way by not plotting the raw data (likelihood of cheating vs belief in a just world), which would show a very small difference. This is yet again a case of "we found a statistically significant but likely meaningless and unlikely to be replicated difference".

> The authors deftly hide this in the standard way by not plotting the raw data (likelihood of cheating vs belief in a just world), which would show a very small difference.

Yet another datapoint to suggest the gp post that academia strongly correlates with dishonesty.

> Yet another datapoint to suggest the gp post that academia strongly correlates with dishonesty.

There's no dishonesty here!

They are reporting data to the standard of their community. We need to raise the level of rigor and convince people to drop the entire concept of statistically significant as meaning anything at all. This all taking far too long unfortunately.

Statisticians made a terrible terrible mistake that doomed science to this bullshit for nearly a century now. People like Pearson and Fisher had good intentions, but the language they used for their methods makes them sound too attractive. So people interpret their methods to be something they are not. We need an overhaul of teaching throughout statistics to excise this cancer now, and that's going to just be very slow.

So no. Authors are doing what they were taught, what is likely best practices in their textbook. It's just that the textbook has been wrong for far too long.

Willing to bet you're an academic.

Hiding the fact that their textual claims don't match their data using various tricks is bog standard academic dishonesty, you see it all the time in anything social or related to public health. It's an endemic problem.

Trying to blame prior generations for defining P thresholds that are "too attractive", or just doing what they were taught, instead of blaming the current authors for doing the wrong things is absurd. No textbook is telling these people to use MTurk or not to draw proper graphs!

The people who compared to the prior study are the authors of this study, right? Shouldn't you take up your complaint with them?

Or rather, don't, because the two studies clearly are comparable. The point of science, whether social or otherwise, is to tell us true things about the world. If the response to comparing the results of two very similar studies is "you can't do that" then this research effort isn't producing generalizable knowledge, which makes it worthless.

Reply on news.ycombinator.com