Comment by _dain_ | Hacker News Reader

Comment by _dain_ | Hacker News Reader

Parent Full thread

>The fact that the DK effect emerges with _completely random data_ is not surprising at all - in this case the intuitive null hypothesis would be that people are good at estimating their skill, therefore there would be strong a correlation between their performance and self-evaluation of said performance. If the data weren't related, then this hypothesis isn't likely, which is exactly what DK means.

DK effect is not that low skill people are overconfident and high skill people are underconfident. It is specifically that low skill people are more overconfident than high skill people are underconfident. i.e. if someone's estimated skill is true_skill+bias+noise, then bias_lowskill > -bias_highskill.

This is very clear in the original DK paper, they specifically focus on the supposed metacognitive deficiencies of low-skill people.

The article argues that the graphs supposedly demonstrating this fact, can also be generated from a model that does not have this difference, i.e. where bias_lowskill == bias_highskill.

EDIT: My characterization of the article is not correct, see here[1] for a visualization of the point I'm trying to make.

[1] http://emilkirkegaard.dk/understanding_statistics/?app=Dunni...

> The article argues that the graphs supposedly demonstrating this fact, can also be generated from a model that does not have this difference, i.e. where bias_lowskill == bias_highskill.

But as I understand it, it doesn't: In the graph generated using random data, the lines intersect in the middle (bias_lowskill == bias_highskill), whereas in DK's paper they intersect in the upper right (so bias_lowskill != bias_highskill).

Okay well in that case the article is deficient (I skimmed it, assuming it was making the same case I already knew about[1] -- my bad). You can actually reproduce the DK graph without supposing that estimation bias depends on skill. There is an interactive visualization here[2] that does precisely that (click on the "line plot w/ centiles" tab). You can adjust the parameters to see how it behaves.

[1] https://news.ycombinator.com/item?id=31038386

[2] http://emilkirkegaard.dk/understanding_statistics/?app=Dunni...

Thanks for the link!

I see what you mean, but is using the "general overestimation" slider actually unbiased? I don't really understand what it is supposed to represent. Looking at the code, it seems to be only an additive term that shifts the plot upwards, but that does not explain anything (actually, in the code, the variable is called "up_bias", and could correspond to an actual DK effect...).

Per DK's hypothesis, the added bias would be a decreasing function of underlying skill, not a constant as it is here. I think this is the only way to define it sensibly. It's easier to see what it's doing if you look at the scatterplot, although it's kind of annoying because the axes are flipped and it keeps the view centred on the data, so the points stay still and the axes move around them. Basically it's just translating all the points uniformly along the estimated score axis.

To be fair, in the random data, you have people of all actual skill levels estimating their ability at 0.

That's going to skew the data somewhat because people don't work that way. While you will likely have some zeros, I wouldn't expect any from the skilled population and I'd expect fewer from the rest of the population.

That's going to make the "difference in estimation" line higher overall, which would make it intersect higher and more to the right.

That would make sense, except that graphs generated from random data show identical bias for overestimation and underestimation, as can be seen in the article. And this is opposed to graphs from the DK paper, which show a smaller underestimation bias for experts than an overestimation bias for low-skill people. (Of course that alone doesn't prove anything, just saying that to my understanding, nothing in the article contradicts my interpretation of DK).

Reply on news.ycombinator.com