The Dunning-Kruger Effect Is Probably Not Real | Hacker News Reader

The Dunning-Kruger Effect Is Probably Not Real

mcgill.ca

70 pointsingve67 comments

The article doesn't explain what is actually going on to produce the second chart, but here is a guess. Assume people have a true ability x, an estimated ability y which is x + some unbiased noise, and a test score z which is also x + some unbiased noise. If you take a sample and collect the lowest quartile of test scores z, then the average of the estimated abilities y in that group is higher than z because the y are produced centered at x rather than centered at z. By collecting the lowest quartile of test scores, you didn't just get the dumbest people, you also got the people who happened to perform badly on the test that day. These people may be estimating their ability accurately, in which case the estimate would be higher than their randomly bad performance that day.

This seems like too obvious a mistake to not have been noticed for this long though.

So it's just artifact of testing.

Lowest test scores come from low ability plus bad luck and as such are lower than just low ability and highest test scores come from high ability and great luck and as such are higher than just high ability.

Easily verified by testing more. (Via law of big numbers.)

Unless the test is biased which means we're back to square one - discerning bias of the test from bias of respondent.

To do that you'd have to produce a specifically biased test in a different way, see how the noises add.

Using quartiles like this is probably ill advised. Just show the scatter.

However, from my reading of the article they were asked to estimate their performance after they had already taken the test, not their inherent ability.

Does this article actually contains explanation of what was wrong with the original study?

Maybe I don't know how to read with understanding, but to me this article makes only two points:

1. The effect was about all people, not just dumb people.

2. The same graph can be obtained "by random".

Point 1. makes sense, but 2. is left unexplained. There is a lot of words, but no content. Almost as if GPT-3 was writing it :) .

Yea I agree. It all relies on (2) but they do a very poor job explaining what exactly the simulation is. If there is a paper by McNight, it isn't cited and I can't find it. Can't find it on his webpage[0][1] or google scholar.

This is a non-trivial simulation so the details are really important! Shame they are nowhere to be found.

[0]: https://ibi.gmu.edu/faculty-directory/patrick-mcknight/ [1]:https://scholar.google.com/citations?hl=en&user=sH44LC4AAAAJ...

Exactly! Why does the author not tell us how the McKnight model works?

This is the only substantial claim the author makes, yet the result is delivered in two measly sentences.

I feel snookered.

The other paper by Ackermann also has a simulated graph, but I have yet to find the simulation model.

On top of that, the graphs are only barely similar, and I would not call them the same. There is a clear difference in magnitude.

The article gives us a few clues:

1. the DK-effect supposedly increases the noisier subjects estimates of their abilities are, 2. the graph relies on breaking the data into quartiles 3. experts are less noisy in their estimates of their abilities than are non-experts.

There is also the paper: https://scholarcommons.usf.edu/cgi/viewcontent.cgi?article=1...

Still I think the article does a disservice in not laying out the argument in plain language. I don't particularly feel like puzzling it out

Point 2 also had me scratching my head. The supposed evidence for it (that graph) is statistical mystery meat.

If the two metrics (confidence and ability) were independent in a simulation, you'd expect to see a more-or-less flat line on confidence if you sorted by ability. So the model clearly insists on some sort of association between them, no? If the association is weak, you get a cross-over. Big deal.

Also, the graphs don't really look all that similar - especially when you consider the fact that the line on which the associated variable have been sorted by quantile is always going to be ascending (duh).

Tufte says "to clarify, add data" - one thing that neither the chart from the original study nor the simulated one respects. Surely we'd learn more from also being able to eyeball a scatterplot.

As I understand it, the original study suggested a bias, which causes us to assume that our abilities are closer to the mean distribution than they actually are. The example about the language test misses out on this entirely.

(Regarding that example, a person may even withstand the confrontation with the test score, like, "I guess, even a native speaker doesn't get more than 65% on average", or, "99% is surely not that unusual, everybody with a decent understanding should get a score like this".)

Personally speaking, it just struck me as odd that - outside of some rare exceptions - that someone could be so dumb that he'd actually be ignorant of his ineptitude. This is different than misrepresenting your ability. Lots of people do that for lots of reasons, but for someone to be so sheerly ignorant as is often given in the case examples struck me as someone who will sooner rather than later learn painful lessons about the truth of his own shortcomings.

The people I know who fall into this category construct an elaborate world view where they are the constant victim of all the idiots and jerks of the world. Everything that goes wrong is someone else’s fault, not their own ignorance and bad choices.

So half of reddit?

Maybe the HN delusions of grandeur effect is real after all

(...I like both, merry Christmas)

I'm not saying reddit doesnt have any uses. But /r/popular is not what it once was. Like imagine if HN was covered in memes and low quality content.

You're grossly underestimating

Its the DK effect in action.

In other words, the Dunning-Kruger effect is a by product of deep narcissism (as opposed to sheer stupidity or ignorance, making it a palatably sinister pathological fungal outgrowth, truly scary).

They say these anti social personality disorders only represent 1-2% of society, but I always try to look at the results of a society to infer the true percentage.

I’d say Nazi Germany had a particular anti social personality disorder at scale, over 50%.

I’ll let you guys infer where everyone else is at now days.

Every single relationship you’ve ever had has failed and it has always been the fault of the other person? Please tell me more.

You might find this study interesting:

"Researchers identify a new personality construct that describes the tendency to see oneself as a victim"

https://www.psypost.org/2020/12/researchers-identify-a-new-p...

I think it's generally understood that people have varying degrees of self-agency. If you're the sort who doesn't believe in your own agency, that everything happens to you rather than you being largely in control, then when life hands you lemons you will consider yourself a victim.

IMO, like many things interpreted at a surface level, it’s easy to miss the detail.

“Dumb” is the wrong descriptor. If you’ve ever worked in a politicized environment, it’s easy to observe spectacularly talented people who are uniquely unqualified to do whatever they are doing. Those are usually good at something else, and those skills generally allow them to avoid consequences for their incompetence, unless they get unlucky.

Another common scenario you see in business is when attorneys get put in charge of things, or talented salesmen get put into non-sales executive jobs.

It's really not uncommon. You can go a long time before you have an accurate self-assessment forced on you. https://youtu.be/jq1_2P6YWtA?t=259 is good, a typically honest reflection from Anthony Stauffer, someone who is no idiot.

Not sure if the Dunning-Kruger effect can be applied to everything (such as: humour, grammar, etc., as listed in the article), but I think pretty much everyone who has played any kind of video game (Dota, LoL, Star Craft, CS:GO, etc.) in a competitive "try-hard" environment for long enough will assure you that the Dunning-Krueger effect is indeed real.

"It's not my fault for losing this; I am clearly better than the team/oponents if it weren't for bad teammates/cheese tactics/meta abuse/lags/etc" is a mindset a lot of players have even if all evidence points to the contrary (~50% winrate in average in matches with players of similar ranking etc.).

In fact, I struggled quite some time with this myself - acknowledging and understanding one's own shittiness is one of the very first steps of actual improvement, and that's usually easier said than done, at least for me personally.

Same goes for IT projects, junior developers usually have a significant confidence when facing new problem (task estimation). Their confidence sometimes comes from limited ability to see all the path that may go wrong, on the other hand senior developers although have considerable experience, sometime overthink solution because of the past experiences, and often they leave extra space for so called "known unknowns" and "unknown unknowns".

While I've certainly seen players say those things, and agree that increasing humility is a good corrective to that character flaw, that kind of public bragging isn't indicative of their self-evaluation, and I haven't seen that expert players necessarily stop that behavior or demonstrate that humility was the key to their skill mastery.

But you need to differentiate what people outwardly project is their ability and what they actual think is their ability.

And then there is also the reaction in the moment vs the reaction after sometime of thinking about it.

There wasn't an actual explanation as to WHY it wouldn't be real. They show the original plot, then another plot generated with random data that looks like a dampened version of the original effect.

Then there are like 10 paragraphs on either side of these two pictures, which don't do a lot to support these critical plots. I am left unsatisfied!

> "The above Dunning-Kruger graph was created by Patrick McKnight using computer-generated results for both self-assessment and performance. The numbers were random. There was no bias in the coding that would lead these fictitious students to guess they had done really well when their actual score was very low. And yet we can see that the two lines look eerily similar to those of Dunning and Kruger’s seminal experiment. A similar simulation was done by Dr. Phillip Ackerman and colleagues three years after the original Dunning-Kruger paper, and the results were similar."

OK, so you removed the brain, and made some models that produce somewhat similar results. What if your models just modeled the brain? How does this refute the original?

This code seems to somewhat re-create effect, except noise is symmetric at both sides (also true of the xkcd-style plot in this article, actually). Plot: https://imgur.com/a/dGZyylf

It seems a crucial piece of context is that there is some correlation between perceived and actual performance. There also is a global optimistic bias, apparently across experiments people perceive their rank is in the 66th percentile, so this universal effect leaves more room for error on the under-performing people while the highly-performing people are going to end up closer to that 66th percentile tautologically, and bounded above by smaller amount of space.

  import random
  import numpy as np

  np.random.seed(seed=12345)

  from scipy.linalg import eigh, cholesky
  from scipy.stats import norm
  from matplotlib import pyplot as plt

  #Draw correlated random variables
  #Ref here: https://scipy-cookbook.readthedocs.io/items/CorrelatedRandomSamples.html

  num_samples = 125 * 4                 #
  x = norm.rvs(size=(2, num_samples)) # uncorrelated random normal variables

  expected_correlation = np.array([[1.0, 0.19], [0.19, 1.0]])
  c = cholesky(expected_correlation, lower=True) #there's a slight correlation of R = 0.19 between actual and perceived scores (according to Ackerman, 2002)

  trans_x = np.dot(c, x)

  #perceived / actual readings with R=0.19
  perceived = trans_x[0, :]
  actual = trans_x[1, :]

  #Sort both variables by actual scores.
  sort_by_actual = sorted(range(num_samples), key = lambda idx: actual[idx])

  perceived_by_actual = [perceived[i] for i in sort_by_actual]
  actual_by_actual = [actual[i] for i in sort_by_actual]

  quartile_indices = [i * (num_samples // 4) for i in range(5)] #note: depends on divisiblity by four

  x_coords = [xx // 2 for xx in quartile_indices[1:]] #mid-points just for plotting
  perceived_means = [np.mean(perceived_by_actual[start:end]) for (start, end) in zip(quartile_indices[:-1], quartile_indices[1:])]
  actual_means = [np.mean(actual_by_actual[start:end]) for (start, end) in zip(quartile_indices[:-1], quartile_indices[1:])]

  #Plot
  fig = plt.figure()

  ax1 = fig.add_subplot(111)
  plt.title("Dunning-Kruger")

  ax1.scatter(x_coords, perceived_means, marker="s", label="perceived")
  ax1.scatter(x_coords, actual_means, marker="o", label="true")
  ax1.legend()

  plt.show()

Could the article actually getting around to explaining the simulation using random data? Frustrating to read the things, and then just be shown a graph purportedly from random data and not explain how it was generated. Is it an artifact of using quartiles?

We all at least are dumber than we'd like to be. Nobody would refuse a pill that would make them smarter. Sorry for off-topic, but this has been gnawing at me lately, feels like something is a miss about my cognition, only very subtle, nothing to go for any kind of practitioner. Is there any route actually for giving the good old fat bucket a rattle to make it work back at original capacity? Have you ever tried anything?

I know this is off-topic, if it annoys anyone, I could take it somewhere else, but thought people interested in this article might have experimented or self-reflected.

>[...] both experts and novices underestimate and overestimate their skills with the same frequency.

Looking at these graphs there is maybe some curious conclusion to be made about the symmetry with imposter syndrome.

If we believe to have 'average' (2nd/3d quartile) knowledge about a subject, there are three possibilities:

1. We are correct in this assumption.

2. We are victims of Dunning-Kruger and know less than we believe we do.

3. We are victims of imposter syndrome and know more than we believe we do.

> instead showed that both experts and novices underestimate and overestimate their skills with the same frequency. “It’s just that experts do that over a narrower range,” he wrote to me.

This makes sense, in the narrow definition they use for the effect, that is students estimating their exam score.

It doesn’t necessarily disprove other interpretations that people are casually using, such as imposter syndrome, which are related but pedantically not the same effect.

I think there are excellent intuitive arguments in favor of the Dunning-Kruger effect.

One is that university students often talk about how suddenly ignorant they feel once they start taking classes with depth of content — and the Ph.D. students who say the feeling only increases the deeper you go.

1. Dunning-Kruger effect is typically defined as X. 2. Here, we define it as Y. 3. We show that Y is false. 4. Therefore X is false.

Uh, no.

At the very least, I wish people would look at those graphs, and realize that the popular “mount stupid” formulation is basically fan fiction.

In my experience Dunning-Kruger 100% is there but in the following form: "knowledgeable people doubt, and less people know they are more confident." It is even amazing sometimes, how confidently people without any knowledge talk and give opinion about subjects that need high level of knowledge, just to even begin to ponder about it. And again, it can be my bias.

I don't know why your response got down voted. Maybe some rather specific people got their panties in a twist.

Either way, what you said is something Bertrand Russel (certainly not best known for being a dumb man) once worded like this:

"The fundamental cause of the trouble is that in the modern world the stupid are cocksure while the intelligent are full of doubt."

.. and just for good measure, here is another one of his quotes, which is hardly well know to those who would probably be better off if they did:

"Our great democracies still tend to think that a stupid man is more likely to be honest than a clever man, and our politicians take advantage of this prejudice by pretending to be even more stupid than nature made them."

In summary: ignorant people do appear to often be more confident than they deserve to be. Whether that is actually because they are truly unaware of their ignorance, that could certainly be a good question to ask oneself. For there are plenty who deliberately abuse both faked ignorance and confidence just to bamboozle and deceive others.

Someone said "if you wouldn't attack yourself in street fight don't do it verbally", but for the sake of good British joke I will say following: So, down-voting my answer is confirming my statement. I do not know much about subject but I confidently claimed 100% effect, therefore confirming what I said :)

It is like that brain melting sentence "If I say that I am lying, am I lying?"

If I claim that Dunning-Kruger Effect is 100% real is it really real? :)

Marry Christmas and Happy Holidays everyone!

> I do not know much about subject but I confidently claimed 100% effect, therefore confirming what I said :)

Well, that's not a Dunning-Kruger Effect either, but maybe a confirmation bias mixed with survivorship bias.

Pffffffffff *amn it ... I can not be right about being wrong in a proper way ...

My rationalization about that effect was around the idea that the deeper is your knowledge on a topic, the more aware you become of how much is left to know. Turning what you don't know that you don't know into something that you know that you don't know.

Not being aware of things that you don't know gives you confidence.

Wait a sec’. The author admits the effect has been confirmed by others, then claims it’s probably not real because of a pair of studies (both from the same team) that even the authors admit support at least part of the Dunning-Kruger hypothesis (“It’s just that experts do that over a narrower range”). His confirmation is a couple of ad hoc unpublishable investigations? It’s almost like he might be overestimating his ability to evaluate the work. In particular, he seems to be more interested in refuting a popular perception of what the effect is rather than the more subtle claims that Dunning and Kruger actually made—another source of the type of errors predicted for those without expertise in a given field.

Of course, an attack on Dunning Kruger demonstrating questionable competence to evaluate their work is delightfully meta.

It's most definitely real

The supposed smoking gun in the article is that if we plot random values for perceived and actual ability we get a chart that looks a lot like the chart in the original study.

I don't see how you make the leap from there to saying that the Dunning-Kruger effect is probably not real. The headline is totally overblown if you ask me.

I guess they could just be more epistemically modest and say that the data doesn't support the existence of the Dunning-Kruger effect, but doesn't make it not real.

My goodness, two downvotes for distinguishing between a failure to show something exists and proof that it doesn't exist.

If you claim that you have meaningful pattern in the data, but someone replicates your pattern with meaningless data then it's perfectly valid to argue that your pattern doesn't come from the data but rather from how you chose to process the data.

> The darling of those who wish to explain why incompetent people don’t know they’re unskilled, the Dunning-Kruger effect may actually just be a data artefact.

This doesn't strike me as true. Or rather, I don't think that it's a significant observation that some people may want to try to explain why some people are incompetent. In my experience, people don't try to "explain" incompetence in general. They will rather talk about specifics - e.g. "Some people believe this because of the misconception that X implies Y". Even if their own rationale is incorrect.

This is very different to saying that "They are stupid because they don't know that they are stupid".

Yeah it may come up as a flippant comment about an individual (or "fringe" group like the Flat Earth Society) in casual conversation, but when I consider how much I've read about and discussed the DK effect, I can't say that the implications as stated in that the quoted piece actually has a significant basis in reality. In other words, DK may be the "darling" of said group, but the self-same group is not very significant in any way in my experience.

YMMV.

I can't go a week without seeing people mention Dunning-Kruger on Reddit. It's become such a pop-cultural zeitgeist term that it's lost any ties to the original study. It's frequently abused as a derogatory adjective to describe people while claiming the superior argument, often without providing evidence.

I don't know what to do about it, but its use by lay people bothers me. It's ironically Dunning-Kruger.

This is a popular opinion on HN, but I'll state again that reddit is basically unusable. I still check a few subreddits using old.reddit.com but the mobile app they constantly shove down your throat is bloated trash and the redesign is garbage. It's inevitable that something better comes along.

You can try to explain that the effect doesn't actually apply to any pointing out of the effect in someone's comment, maybe even cite the study.

Agreed. And its use in discussions unfortunately adds no value. Imagine that two people are arguing about something. You walk up, interrupt them, and state that one of them is wrong but because the wrong person is stupid, they don't realize they are wrong.

Will your comment change anything? Of course not, both people arguing will just assume that the other one is the stupid person.

I can't help but notice that your comment ironically continues the recursion chain, and that mine now does as well.

Isn't the Dunning-Kruger effect about personal mastery, and not comparative social science?

Not enough evidence in the paper. And does anyone feel like this effect is not real in this year of all years? People were screaming 'fake news', 'fake virus', 'fake election', blah blah with great conviction.

Search HN for the effect, you'll see criticism of people citing it from years back, so it's not this year's thing at all.

As I say, "Citations of Dunning-Kruger are usually examples of it."

Gave me a good chuckle

funny recursion (if intended)

I'm sure the Dunning-Kruger effect is real, and I'm much better than average at being able to tell!

To me it ends up feeling like people who say things like "people are stupid" and use the Dunning-Kruger effect to hand wave away harder socio-psychological problems are projecting their own insecurities of their lack of knowledge and can be safely ignored

The Dunning Kruger might be a mirage but this article doesn’t seem to show that.

It shows 2 charts, one that was used by Dunning Kruger and the other generated by random noise.

Both look similar, but in the random noise chart the 2 lines intersect midway along the x-axis, where the 2nd and 3quartiles meet, while the dinning Kruger chart has the line intersecting where the 3rd and 4th quartiles meet.

Isn’t that a significant difference between the 2 results?

Reply on news.ycombinator.com