Comment by jldugger | Hacker News Reader

Comment by jldugger | Hacker News Reader

> Again, my main point is that there’s nothing inherently flawed with the analysis and plots presented in the original paper.

I find the use of quartiles suspicious, personally. It's very nearly the ecological fallacy[1].

> I’m not going to start reviewing and comparing signal-to-noise ratios in Dunning-Kruger replications

DK has been under fire for a while now, nearly as long as the paper has existed[2]. At present, I am in the "effect may be real but is not well supported by the original paper" camp. If DK wanted to they could release the original data, or otherwise encourage a replication.

[1]: https://en.wikipedia.org/wiki/Ecological_correlation [2]: https://replicationindex.com/2020/09/13/the-dunning-kruger-e...

Agree. From the DK article graph it is not possible to separate the cases

1. Average self assessment coincides with true skill, but variance increases with low skill.

2. Average self assessment is biased, and the bias is positive when you are unskilled and negative when you're highly skilled.

These two situations would create indistinguishable DK-graphs. I don't understand how anyone can be sure on either (1) or (2) after seeing one instance of such a graph.

As I see it, the only way out for "DK positivists" is to say that the DK hypothesis is unrelated to the truth values of (1) and (2). Or, that there is other evidence making DK convincing.

Neither seems very plausible!

FWIW extreme groups (e.g. using upper and lower quartiles) is well understood in its inflation of effect size (there are even formulas to correct this, given an extreme groups design).

It's definitely related to ecological fallacy in the sense that both underestimate relative error and inflate effect sizes.

Do they have to encourage replication?

If others can't replicate it entirely on their own without "encouragement". Then it isn't useful at all and the original experiment can be safely ignored as irrelevant to humanity, along with any "prestige" associated with it.

Reply on news.ycombinator.com