Misunderstanding Dunning-Kruger | Hacker News Reader

Misunderstanding Dunning-Kruger

theness.com

169 pointsimartin2k86 comments

> if you think it only applies to other people (which itself, ironically, is part of the DK effect) then you miss the core lesson

I have observed in conversation that invariably if someone brings up Sturgeon's Law [1] either by naming it or quoting it ("90% of everything is crud|shit|garbage"), they invariably place themselves in the 10%. As if awareness of the law somehow insulates the speaker. That seems to be the case with D-K, per this article.

[1] https://en.wikipedia.org/wiki/Sturgeon%27s_law

Moreover, the original point of Sturgeon was not a simple dismissal of "90% of everything" but his dismissal of any informativeness in somebody's similar claim for anything specific (he wrote about the art forms):

"I repeat Sturgeon's Revelation, which was wrung out of me after twenty years of wearying defense of science fiction against attacks of people who used the worst examples of the field for ammunition, and whose conclusion was that ninety percent of SF is crud. Using the same standards that categorize 90% of science fiction as trash, crud, or crap, it can be argued that 90% of film, literature, consumer goods, etc. is crap. In other words, the claim (or fact) that 90% of science fiction is crap is ultimately uninformative, because science fiction conforms to the same trends of quality as all other art forms."

So you're saying 90% of invocations of Sturgeon's Law are crap?

Of course, to follow Sturgeon's own line of thought, that doesn't mean the other 10% aren't worth taking note of.

I have never heard Sturgeon's Law invoked to place the speaker anywhere: I've only heard it as a rebuttal of a dismissal. If someone haughtily says they don't watch TV, and cites three shows as reasons why TV is terrible, well, yes 90% of TV is crap, but so is 90% of opera, 90% of stage plays, 90% of tweets. If we judge anything based on the median, it's crap, so it makes more sense to judge things based on their best examples.

And if 90% of invocations of Sturgeon's Law are crap, well, we should judge them by the best examples too :)

I hazard to propose that twitter contains closer to 99.999% crap. 90% might be the central point, but I think there is still variance between different mediums. Comparing nominal values without knowing the distribution can be dangerous, but that's why we have units of deviation, such as zscore.

The best part of Sturgeon's Law for me was realizing that it's tautological. It's just defining "crud" to be the bottom 90%, which is drawing an arbitrary line between the top and the bottom. It's a fun one in the same way that D/K is in that it just feels true, like it's validating a working theory you already had about the world.

But Sturgeon's law can't be false, whereas D/K really is commonly misunderstood and might not even be generally reproducible; it might not even be broadly true at all. D/K is certainly over-generalized and in popular culture applied to situations that are very unlike the experiments in the paper, which were only pretty trivial tasks. The D/K Effect "actually goes away, and even reverses, when the ability tests given to participants are very difficult." https://www.talyarkoni.org/blog/2010/07/07/what-the-dunning-...

I dislike when people use adages like these (___'s law). They rarely add anything to the conversation and I think a lot of confirmation bias is involved. Moore's Law is an exception to this because it's actually falsifiable and less of an adage.

> The best way to summarize the data is to say that the difference between self-assessment and performance increases with lower performance below about the 80th percentile, with slight underestimation of performance for those above the 80th percentile (or a test score of about 70%).

It may just be me, but this does not seem like the best way to ‘summarize’ the data for general consumption.

The original “dumb people always think they’re smarter than they actually are” that the author takes issue with seems more sensible.

We can’t expect everyone to be in the top 20%.

> dumb people always think they're smarter than they actually are

This is a really bad summary of the original D/K paper, both misleading and incorrect. Maybe this illustrates nicely why the article was needed. They didn't knowingly test any dumb people; it wasn't a random population sample, it was all Cornell undergrads volunteering for extra credit. The results are relative, not absolute, they were being asked to rate themselves against the population (their classmates), not actually rate their ability. That involves guessing how good others are. We don't know how good they all were in absolute terms, only their relative scores. The results did not conclude anything close to "always", it only showed an average bias. The results show that, on the tasks they tested, the people with the best skills under-estimated their scores relative to their peers.

the results are indeed relative. the smarter one gets the same thing still applies. it's a theory not to point at others and say 'he's on mount stupid'.. but rather one to be wary of yourself being on this curve, which in some sense or aspect one always is, as one is never a master of everything in their lives. it's a theory about evaluation of self only applicable in that domain.

I wonder if that's because "best skills" entails "most suspicious" too.

> “dumb people always think they’re smarter than they actually are”

But that makes it sound like only people at the bottom overestimate themselves, when in reality 80% of people overestimate themselves. How about this rewrite: "People who aren't experts think they're smarter than they actually are." And get rid of the word "always", we're talking about averages, not about certainty in every case.

I think you're falling into exactly the same inaccuracy that you're trying to combat. The issue is not that everyone below the top 20% of performance over-estimate their performance, it's that over-estimating performance is strongly correlated to incompetence. Your summary makes it sound like someone who is of average competence is over-estimating their performance to the same extent as those with the worst performance, which is untrue. In fact when we talk about the DK effect almost no one is talking about the average person who thinks they're smarter than 65% of other people because that effect is almost impossible to qualitatively observe.

I agree, not unimportantly the effect is much more pronounced in the bottom group.

When someone overestimates their abilities by 20% it is far less drastic than a whopping 500%

500%? I don't think you can compare percentiles like that.

Let's say someone estimates his percentile to be the 3rd percentile (aka 97% of people are better) and in reality he's in the 1st percentile (99% of people are better). Did he overestimate his ability by 200%? I wouldn't say so, I would say he was very very close.

Even at non-extreme percentiles I don't think such a comparison works. For example in some area it might be that people at the 10th percentile know nearly the same amount as people at the 50th percentile. If it's some area that most people know little about. So the abilities of someone at the 10th percentile would be nearly the same as the abilities of someone at the 50th percentile. So overestimating your percentile by a lot is not the same as overestimating your abilities by a lot.

What I see in practice is that the effect is often used to discredit the competency of people like the WP author did.

> which itself, ironically, is part of the DK effect

I think the effect can help you come to the conclusion many experts in fields come to after a while: The larger your knowledge, the more you are aware of your ignorance. I am sure there is an English idiom that describes it more gracefully. Maybe that is the intersection to the 20%.

Enthusiastically pointing out the discrepancy on the other end of the curve is most probably an indicator that you might have fallen "victim" to the effect itself.

It reads like no one had any sense of how well they would do on the test. So people guessed they'd be slightly above average which is a good prior for a group of Ivy league students. This lead to the lowest being "overconfident" and the best being "under confident". Designing good studies is hard.

This article was surprisingly OK.

About all it omits is that people at the bottom of the scale generally didn't even understand the questions they were asked, so their answers were really all over the place.

But the key takeaway -- the D-K effect is about you, much of the time, even if you are expert about some things -- is usually neglected. D-K is about everyone: The more certain you are, the less likely it is you have reason to be.

But, judging from the graph in the article, the top quartile actually under-estimates their ability? Only lower 3 quartiles overestimate it, so for experts it doesn't seem to be true? Unless, of course, I misunderstand DK effect completely and am not aware of it...

Yes, the point is that in your area of expertise you are likely to be more humble relative to your actual ability (but not too far off, on average).

However outside of your specific area(s) of expertise your confidence in your ability is likely to far exceed your actual ability (on average).

So experts are still susceptible to DK effect for areas that lie outside their expertise. Even (or perhaps especially) if they think their expertise gives them special insight into another field.

This makes sense to me. If you're an expert you are comparing yourself to other experts in your field. On the other hand, if you know anything about a topic but only interact with people who know nothing at all about it, you would rate yourself highly.

For example, I read a lot about modern physics but am not a physicist. I would give myself an above average understanding but would likely fail an exam.

It’s true for experts outside of their field of expertise.

It's true for experts (above ~70th percentile) in their area of expertise, too, just in the opposite way as it is true for people below that level.

The more you know, the more you know that you don't know...

Cloud computing has been a DK bulldozer.

Time and again the symptoms of the problem I'm troubleshooting have correlated with some prior experience, triggering a prediction, which has been wrong.

And I've actually passed multiple cloud certifications, too.

IT is just complicated.

My initial hypothesis on bug fixes are normally wrong. However proving/disproving them tends to help immensely in finding the real problem.

My DK assumption/bias is that people who tout their “multiple cloud certifications” are likely more wrong than average engineers.

Would be interesting to find out how wrong I actually here...

Well, those AWS Associate certs ain't beanbag.

However, if you haven't had the service blow up in your face a few times, you'd better have a photographic memory to pass those certs.

> the D-K effect is about you, much of the time, even if you are expert about some things

It's as much about you and the things you are an expert about as anything else; D-K isn't just a low-ability effect.

No, people under-rating their expertise is a garden-variety variation on reversion to the mean. D-K, if it means anything, is an effect on top of that.

> No, people under-rating their expertise is a garden-variety variation on reversion to the mean.

Reversion to the mean would be underperforming past elite performance, not underestimating actual performance.

And it would draw toward the mean, not the ~70th percentile.

> [T]hose who endorse anti-vaccine views were far more likely to estimate that they knew as much as experts or doctors, when in fact their knowledge level was extremely low.

> Dunning believes that ability to self-assess is also key to the effect.

> As you gain knowledge, this confidence grows.

> Specifically, narcissistic personality seems to put DK on steroids.

That collection of quotes feeds my suspicion that extreme examples of the DK effect and conspiratorial thinking (For example: vaccine-denial, birtherism, racism as "expertise" on how the outgroup "really is," and so on.) have in common the fact that they give those inflicted a much-needed sense of control over an uncertain universe. Baseball players are known for having superstitious rituals at the mound or the plate, precisely because pitcher and batter are engaged in high-stakes, high-variability activities with unpredictable success rates. Baseball players are hardly traumatized, but imagine if they were playing for their lives, or their senses of self.

I wouldn't be surprised if the fractured way that the world is presented to us after the decline of television and the rise of the internet (with its conflicting narratives) hasn't so traumatized a fraction of the population that the only way they can construct their sense of selves as in control enough to not feel permanently precarious is to habitually steer themselves away from even the hint of dissonance by constantly asserting, "I know. I am in control. This is how it is."

One of the things having one or a few sources of national news possibly did was to give one the plausible belief that their set of beliefs was tenable, stable, and unchallenged by rivals. From a scared monkey perspective, I wonder if we got lucky with broadcast television, if the perception of unity wasn't necessary. Whereas now we seem to be constantly having to defend our perceived in-group.

The point of this ramble is that maybe the psychodynamics of DK are as simple as, 'Feeling confident feels better, and I need that, and I feel rewarded when I engage in thinking that rewards that sense.'

I've always thought Dunning-Kruger is kind of bullshit. (I realize the irony in stating this.)

In the study a bunch of people took four tests and then estimated their score RELATIVE TO OTHER TEST TAKERS. That's bullshit!

The test takers were 65 Cornell psychology undergraduates. Cornell is a great university. All 65 test takers are probably super smart! In fact, that might have all scored quite similarly. Forcing 65 high performers to estimate their relative score is bullshit. That's impossible! You don't have enough information about the other test takers.

I think it would be far more interesting if test takers estimated their absolute score. Whether that score is better or worse than other test takers is irrelevant.

But what do I know. I'm probably just incompetent and don't know it.

well given the https://en.wikipedia.org/wiki/Replication_crisis I think it's healthy to question psych experiment results. I think you'd want good evidence that this effect can be measured repeatedly and in different contexts.

The point of this article was too remind us that just because someone is smart in one area (Cornell psychology undergraduates) doesn't mean they are smart in another (humour was measured in original paper - have you read the DK paper?).

The article makes the common mistake of ignoring the other side of the DK effect: the top quartile estimates their skill downwards.

I like to say I am showing DK tendencies because I often underestimate my skills ;-)

My unpopular opinion is that the majority of people who mention Dunning-Kruger or imposter syndrome are primarily just trying to humble brag.

Full Duning-Kruger paper is here. https://pdfs.semanticscholar.org/e320/9ca64cbed9a441e5556879...

There is some commentary on raw test score estimation. Including this funny line:

"Participants in the bottom quartile also overestimated their raw score on the test. On average, they thought they had answered 5.5 problems correctly. In fact, they had answered an average of 0.3 problems correctly"

I wonder if the raw DK data is available? The paper includes a small handful of figures. There's quite a few more I'd like to render.

> (I realize the irony in stating this.)

Nah, if somebody called you out on it they'd be begging the question.

My pet peeve is people thinking that D-K is a negative correlation of ability and perceived ability, when it's actually a weaker correlation than it 'should' be.

And due to scale-end effects, you should expect to compute a weaker correlation from a naive calibration analysis. If you are at the zeroth percentile, you can only overestimate your performance; if you are at the hundredth, you can only underestimate it. http://home.cerge-ei.cz/ortmann/TrentoCourse/Juslin_etal_Nai...

But interestingly, D-K isn't a symmetric-about-median effect; which is what you'd expect from scale-end effects alone. The D-K finding was that people tend to estimate themselves closer to roughly the 70th percentile than they actually are.

70% is the bottom of C grade range. (https://en.m.wikipedia.org/wiki/Academic_grading_in_the_Unit...) Honestly I think this can explain a large part of D-K. People just aren't calibrated to imagine getting a 25% let alone a 0% even in something they are truly terrible at.

> 70% is the bottom of C grade range

70th percentile doesn't correspond to a 70% score, and when it does, it's almost certainly not on a (common, but very far from universal) grading scale where it is true that 70% is the bottom of the C range, just like 50th percentile is very rarely failing, though 50% is in (and not even the top of) the F range on the same scale.

> Honestly I think this can explain a large part of D-K.

It almost certainly doesn't explain any of the high-end part of the effect, and an explanation where the low-end and high-end effects are unrelated coincidences is, while possible, the kind of explanation that a preference for parsimony would prefer avoiding in the absence of evidence demanding separate effects.

I just looked at the original paper again and I see that in some of the tests the researchers asked participants to estimate the raw number of questions they got correct, in addition to the percentile relative to their peers. Even in raw numbers the bottom quartile overestimated their ability significantly. So I'm less confident in my theory than I was before.

Where are you getting this from? The actual test scores in the graph in the article don't correspond with the percentiles; I don't think the scales on the x and y axis are the same, which is what you seem to be contending.

In an eating contest, it's rare to finish eating less than one bite. It's not insane to imagine that last place is significantly better than 0.

Even for percentiles, it makes sense to calibrate to above average: people are rarely ranked on tasks they're not good at.

But that’s sort of what you’d expect, e.g. most people thinking they’re an above-average driver—but don’t actually think they’re incredibly wicked good. I’d never seen it presented quite this way before but it makes a lot of sense.

Nobody said it had to be scale-end effects alone. But yes, when I look at that Dunning-Kruger graph, I don't see what the authors of TFA say or what the common misinterpretation is (I agree with the authors about how it's commonly misconstrued), I see bad calibration, but the rank order of everything looks right.

D-K also stands for Dorling-Kindersley, publishers of wonderful photographic books which encourage the D-K effect, which is wanting to buy more. Also, create false sense of understanding of the "aha, thats what a triceratops looks like" form, leading back to this D-K effect

I highly recommend the author's podcast: The Skeptics' Guide To The Universe [1].

[1] https://www.theskepticsguide.org/

The women were much more likely to turn down the opportunity: only 49 percent of them signed up for the competition, compared with 71 percent of the men. “That was a proxy for whether women might seek out certain opportunities,” Ehrlinger told us. “Because they are less confident in general in their abilities, that led them not to want to pursue future opportunities.” -Dunning and Ehrlinger

https://www.theatlantic.com/magazine/archive/2014/05/the-con...

My takeaway from DK is that the roughly 80% mark is where is the generally ideal balance between self-image and social acceptance, and that's why everyone on either side leans toward that inflection point.

Having been on the "exceptional" side of the curve on some things (and verified though both independent testing and testimonials), it was definitely socially isolating until I broadened my perspective and accepted that while I may be past that 80% mark in some things, I was well behind it in many others, and if I could appreciate the skill others have that I do not, I would have a much more enjoyable experience of community than comparing others to myself in the things I was the best at.

In general, I think this is a lot of cognitive dissonance & confirmation bias. For everything from politics & religion to what foods or TV shows we like, it's more enjoyable to share our lives with people that enforce our sense of reality. Coming to terms with the fact you suck at something is really unpleasant if you haven't also found the things you are awesome at to balance it out in your self-evaluation. And vice versa, being REALLY awesome at something kind of sucks and makes you feel you can't connect with others (and sometimes they will hate you for outshining them if you shove it in their faces) unless you recognize your own clay feet.

It also explains why echo-chambers like the anti-vaxx FB groups are so popular. They are exceptionally (in the standard deviation sense) out-of-touch with reality, which is typically something that is socially isolating, but by coming together, they can have the benefits of community without sacrificing their identity of being smarter than doctors. It's basically like a reverse-MENSA (which has its own issues with being on the wrong side of DK -- IQ tests are one specific measure of a population variance, and doesn't necessarily correlate with things like empathy, open-mindedness, or creative thinking - all things I'd care about having in my social circle moreso than IQ).

Interestingly enough, we can see the same effect in purely objectively evaluated ranges. Many of us might be familiar with the threshold for how much income correlates with the maximum happiness (on average, having either less or more correlates with unhappiness). IIRC the latest research puts it at about $100,000 times the square root of the household size. While I don't have the tables in front of me, I'd wager that's roughly the 80th percentile in income.

The thing is - that 80% mark is simply the ideal average - but the spread around it is up to us as a society. If we provide better education resources for those struggling in academics, we can get more people closer to the ideal (and likely improve their happiness) without necessarily moving the mean. If we have high marginal tax rates, we can redistribute wealth to close the "spread" around the ideal, which if the research is correct will actually improve happiness for the people on both sides of it. If we actually manage to master our own genes, we can likely do the same for health, innate intelligence, empathy, depression, etc.

Another dimension to consider is that some people perform well at some specific things, but not all nonspecific things. And also that performance varies with situation-to-situation and self-perception varies with mood/time.

In reality, DK is an overall impression tendency rather than a strict rule... such is the nature of squishy, fungible social "sciences;" reality is messier than a numeric IQ result.

I wonder if the DK effect is really an implicit narrowing or widening of the population.

I would guess thay I'm an above-average sprinter. But that's because I'm assuming there are 2 year olds and 80-year-olds in the population. If you ask a runner, they are likely to compare themselves against other runners and probably not even consider someone like me in the population.

I assure you my understanding of D-K is well above average.

I feel like I can't go one day without reading somewhere on the internet where someone mentions the DK effect. If anything, it's a prime example of group think. It sounds clever, and is probably true in a ton of cases, and so people repeat it. Over and over. The same is true of occams razor. These are just ideas, not intellectual vistas.

People love to think they’re really smart and post a Wikipedia link to DK, Occam’s razor, submarine PR, etc. whenever they think it’s vaguely related because they think it makes them look clever. They’re not.

Instagram Intellectuals. Some folks want to show off their hot bods, expensive cars, and exotic trips. Others want to show off just how smart they are with a superficial recognition of a pattern, a concept, a thing they know, but not an actual discussion.

I'm as guilty as any. How does one learn to "shut up" other than through sheer force of will?

Is it actually preferable to shut up though? They say that one of the easiest ways to get a better answer is to be wrong on the internet. Thus if you share what you know, but end up being wrong, then somebody's going to correct you.

Very rarely do people correct you in a way that's instructive. It's usually just mocking after they get you on one tidbit. A large part of that is because those who argue on the internet are usually children or people who know nothing about anything.

It takes wisdom and humility to be a student if your instructor is a fool or sociopath.

Especially since one has an excuse to get indignant and closed minded when that happens.

Hmmm. It takes wisdom and humility to gain an advantage from the networked human culture. And in other cases it makes one dumber or less wise? So plugging in a few reasonable initial values it appears the median wisdom will drop to zero unless a coterie of wise folks are able to partition themselves off in ycombinator or kuro5hin Or Slashdot or reddit.

This was possibly true ~20 tears ago on comp.lang.perl.misc or lkml, but it is certainly not true "on the internet" today, except perhaps in the most marginal of edge cases in places that the eternal september has not destroyed...

Yes. Personally at least, I've found my mistakes aren't particularly unique. But if I'm not the one making a mistaken assertion, I'm less defensive about correcting my understanding.

Practice silence if you are not creating valuable conversation. Pointing out intellectual fallacies ends such discourse, and doesn't accomplish anything but that.

IMO, submarine PR should always be called out. I often times read something, and don't realize its a submarine.

That is the exact thing I like to use HN/reddit for.

What's "submarine PR"?

My searches runneth dry.

The really funny bit is that DK predicts that the people best at a subject will tend to underestimate their own skill compared to the average. That implies they'll overestimate how good other people will be at the subject. Which is the exact opposite behavior of those people trying to look smart.

Confirmation bias is another classic.

Affects everyone but me.

https://en.wikipedia.org/wiki/Bias_blind_spot

Now I look smart, right?

I think René_Girard has some deeper thoughts about this subject (mimetic desire): https://en.wikipedia.org/wiki/Ren%C3%A9_Girard#Mimetic_desir...

I think what is driving the distinction between what the author is claiming and what WashPo is talking about is that whilst it's true the data suggests people gernerally over-estimate their ability relative to others when they under perform, that's not important. It's not important that the average guy thinks he's smarter than 60% of other people. The important bit is that the idiot thinks he's smarter than average.

So there's the DK effect, and there's the interesting part of the DK effect.

But in the original article, the line is drawn to Trump. But I don't think taking a statistical pattern and then applying it to the public persona of a reality TV star is a good idea. If we're going to object to anything, I'll object to that first.

I find it noteworthy that the author seems to be extremely confident that he’s right.

Which, according to the the DK paper makes it more likely that he is in fact right. There is a positive correlation in their results between confidence and ability.

And yet the effect could just as easily be predicting that he is still not confident enough.

I think there's a better explanation at https://www.talyarkoni.org/blog/2010/07/07/what-the-dunning-..., which also discusses some possible alternative explanations.

Some people have a really inflated view of their understanding of the Dunning Kruger effect.

I see on a regular basis people who have poor levels of understanding on a subject and yet believe they know better than the experts.

Climate deniers, anti-vaxxers, anti-GMO, etc. I understand this isn't exactly what Dunning Kruger is talking about, but I don't have a better short hand explanation for it.

When someone invokes the Dunning-Kruger effect, I often reply asking if they are a trained psychologist and are they sure they are using the term correctly? This never goes over very well.

Due to Dunning-Kruger, I don't know if I understand Dunning-Kruger so well or not.

[flagged]

Please don't take HN threads into partisan flamewar.

How does the Dunning-Kruger effect affect people with imposter syndrome?

It just shows that, empirically, most people who think they have imposter syndrome are, in fact, imposters.

Most of the time, I suspect that the "I'm not qualified to be here" feeling is simply normal people having a moment of clarity.

you can’t say stuff like this in public even though it’s true. it’s hurts too many peoples feels

Reply on news.ycombinator.com