Comment by anonymouskimmer | Hacker News Reader

Comment by anonymouskimmer | Hacker News Reader

anonymouskimmer

> If the Dunning-Kruger effect were present, it would show up in Figure 11 as a downward trend in the data (similar to the trend in Figure 7). Such a trend would indicate that unskilled people overestimate their ability, and that this overestimate decreases with skill. Looking at Figure 11, there is no hint of a trend.

There certainly is a hint of a trend. Why do people, when visualizing data with a distinct trend, say that because the "error bars" from a particular statistical test overlap zero that no trend exists!?

Freshman trend to over-confidence. Grad students trend to under-confidence. Undergrads in general trend to over-confidence (though this trend decreases as year in school increases), and post-graduates, whether grad students or professors, trend to under-confidence.

These "trends" are not statistically significant, but they certainly are a trend!

Also, the random data distribution in figure 9 doesn't show the same trends as Dunning-Kruger's curve in figure 2. Perhaps there is at least one psycho-social mechanism here worth investigating?

If they're actually error bars, you can shrink them with more data. That will turn the hint of a trend into an observation of a trend. If it wasn't random noise giving a fake hint.

> If they're actually error bars, you can shrink them with more data.

Assuming the new data has the same systemic or instrumental bias as the old data. Even using a different test date could skew results enough to widen the error bars.

> These "trends" are not statistically significant, but they certainly are a trend!

This is an oxymoron.

Oxymorons only sound contradictory on a surface level.

Something "certainly" being a "trend" is the definition of statistical significance, so this is a straight up contradiction.

See here: https://news.ycombinator.com/item?id=38416858

"Trend" has multiple meanings. Statistics doesn't get to claim all of the meaning.

Show how.

I place mechanistic theory prior to statistics in science. Mechanistic theory can be tested, statistics are a kind of test.

If a statistically-insignificant result shows consistent, though non-significant deviations, such as the kind seen in Figure 11, then it tells me it's worth investigating whether mechanism(s) are explaining a very small portion of the variation that will not, in itself, show up as statistically significant, as it's being swamped by variation in other parameters.

Consistency is a synonym for statistical significance. If there's consistency beyond random alignment, then there should be a statistical test you can apply over your data to extract the signal.

You can extract surprisingly small signals relative to variation in other parameters. But if it's actually swamped, then it might not be real, so go get more data.

> Consistency is a synonym for statistical significance.

So basically you're telling me that if I can visually see a consistency that does not show up in their statistical test, then they aren't running an appropriate statistical test on what I'm seeing.

> But if it's actually swamped, then it might not be real, so go get more data.

Even better to design other experiments.

> So basically you're telling me that if I can visually see a consistency that does not show up in their statistical test, then they aren't running an appropriate statistical test on what I'm seeing.

Either they're not doing the right statistics, or it's a "consistency" that is much more likely to show up randomly than you naively expect, and the study needs to be repeated or enhanced.

Sometimes you can see a pattern that's just a figment of chance. See also: numerology, jelly bean xkcd

Reply on news.ycombinator.com