Dunning-Kruger and Other Memes
danluu.com
danluu.com
edit: I read the article linked in another comment below [1] and now I think I understand what claim the original article was referring to. It seems a common misunderstanding of the D-K effect is that the bottom percentile would think that they are as competent or more competent as the top percentile. However, this is not what the study says, the bottom percentile is overestimating their ability but the estimations are still relatively lower than the self-estimations of those with higher ability.
[1]: http://www.talyarkoni.org/blog/2010/07/07/what-the-dunning-k...
The graphs cited in the article show that the top still estimate themselves better than the lower percentile, which is new to me.
"Rather, it’s that incompetent people think they’re much better than they actually are. But they typically still don’t think they’re quite as good as people who, you know, actually are good."
Unfortunately, while the incompetent people may believe that they are not quite as good as people who actually are good, the problem is that they are not qualified to identify people who are actually good. From the original paper:
"That is, the same incompetence that leads them to make wrong choices also deprives them of the savvy necessary to recognize competence, be it their own or anyone else’s."
"As predicted, participants who scored in the bottom quartile were less able to gauge the competence of others than were their top-quartile counterparts." [2]
So while the incompetent people may not believe themselves to be experts in general, they are less able to identify that the person sitting next to them is an expert.
And further: "Bottom-quartile participants failed to gain insight into their own performance after seeing the more competent choices of their peers. ... [they] tended to raise their already inflated self-estimates."
Not only do they fail to identify the expert next to them, but when presented with the expert's answers they will raise the own inflated estimate!
Therefore, I believe that in practice, the DK paper does support the belief that less competent people believe themselves to be more competent than experts.
[2] Kruger, Justin; Dunning, David (1999). "Unskilled and Unaware of It: How Difficulties in Recognizing One's Own Incompetence Lead to Inflated Self-Assessments". Journal of Personality and Social Psychology
[i]n two of the four cases, there’s an obvious positive correlation between
perceived skill and actual skill, which is the opposite of the pop-sci
conception of Dunning-Kruger.
So, yes, Dunning-Kruger (as evinced from the diagrams sported here) indeed seems to make a weaker claim: that there's no correlation between “perceived ability” and “actual ability”. As such, this claim is as far from the "pop-sci conception" of Dunning-Kruger as it is from the author's.
At any rate, even if we don't take that into account self-assessment is worth plenty as demonstrated by many studies which manage to get coherent data from self-assessments. Sure, you should take them with a grain of salt, and you can expect biases, but no need to throw them out.
Also, what corner of the universe are you from? Loess regression, hierarchical modeling, conditional analyses... methods for finding "non-global" correlations aplenty.
"It’s a little easier to see why people would pass along the wrong story here, since it’s easy to misinterpret the data when it’s plotted against a linear scale, but it’s still pretty easy to see what’s going on by taking a peek at the actual studies."
Googling
http://wws.princeton.edu/news-and-events/news/item/two-wws-p...
"People’s life evaluations rise steadily with income, but the reported quality of emotional daily experience levels off at a certain income level, according to a new study by two Princeton University professors [...]"
So seemingly that has nothing to do with the scale used, but with the definition of happiness. Guess someone didn't look "at the actual study"?
This whole article is just a bag of fail.
I don't think there's a single DK meme, but I do think that this is a common DK meme.
Somewhat-serious suggestion: maybe people who actually understand DK, have a tendency to underestimate how well they understand it relative to everyone else, and so assume that everyone else understands it about as well as they do.
At my previous job where I worked and managed a few young-ish support and jr devs, I came accross a lot of the same attitudes. The funny thing is that two or three years in, they tend to shed those attitudes (at least the smart ones do). They realize that this stuff is a lot bigger than the limited experience they've gotten in school or in their hobbyist projects. Suddenly the "lets toss out everything and do it my way" motor-mouting gets turned down a notch or two, especially after I let them do things "their way" once in a while only to have it explode in their faces. Hell, 90% of managing young techies is controlling their D-K until they mature into imposter syndrome. Then you have to manage that, which is a million times easier to deal because you're not being know-it-all'd to death in every meeting.
D-K is real. The graphs he points out literally supports it. Personally, I'm getting sick of this kind of uber-skeptic mentality that's popular here and on reddit. I think there's something ego gratifying for the INTJ male about being this loud-mouth contrarian who tells everyone they're wrong. Especially if its against some popular wisdom and if the argument is unusually pedantic and trivial.
Hahaha, what about INTPs?
What the author says is: In any case, the effect certainly isn’t that the more people know, the less they think they know. And he's right; the data he shows does not demonstrate that. The more talented people in those plots, at worst, think they know as much as the less talented.
What the graphs do demonstrate is that the less talented people consistently overestimate their ability, whereas the most talented people underestimate. So what's actually negatively correlated isn't actual knowledge and perceived knowledge, it's actual knowledge and the gap between actual knowledge and perceived knowledge.
In conclusion,
Personally, I'm getting sick of this kind of uber-skeptic mentality that's popular here and on reddit. I think there's something ego gratifying for the INTJ male about being this loud-mouth contrarian who tells everyone they're wrong.
Um...
This sort of gratuitous negativity isn't appropriate in a Hacker News comment. It is unnecessary to make your point and pollutes the atmosphere for all of us. Please don't do that.
That's not necessarily money's fault, and I'd point out that there are certainly rich people that don't fall into this category.
People who want to pursue happiness need the freedom to do so. Freedom to pursue happiness can be directly correlated with how much money, or wealth, is available to the person.
It's not money's fault that some people don't know how to be truly happy. Money simply affords freedom.
The problem is that with enough freedom and no idea how to be happy in it, you're going to end up in a worse place than if you didn't have the money in the first place.
Money doesn't automatically make people smug and funless (new word?). People make people smug and funless (when they have the freedom and inclination to (unintentionally(?)) do so).
Happiness means many things to many people. For some, work itself brings them joy whether or not it results in earning a great deal of money e.g. charitable work. For others it is close relationship with their family, a situation that increased wealth often worsens. Many work their entire lives without recognition or compensation to pursue art, giving up comfortable or successful lives and careers in pursuit of something that, more often than not, they are the only ones who see value in. Religious or aesthetic devotions have perhaps the longest history of eschewing wealth and what most would consider a normal life.
As a society, I think that we are beginning to understand the faults in our current model of a successful life and so we've been seeing more of a trend towards simplifying our lives to make room for these other generators of happiness. We are starting to see that the freedom of money is not the same as the freedom from money.
I also agree that we seem to be reaching a better understanding of happiness, thought it's still very fragile and may not last.
The first three I was all like "yes! So glad someone articulate is finally saying this".
Then for type systems I was unable to overcome my current biases, thinking "what a load of tosh! Type systems are obviously a good thing".
Makes me wonder what it would actually take to break the Dunning-Kruger meme
The good news is that this opinion doesn't contradict evidence either! The research just isn't there yet. Some of research I've seen so far is pretty bad.
Key quote:
"I suspect we find this sort of explanation compelling because it appeals to our implicit just-world theories: we’d like to believe that people who obnoxiously proclaim their excellence at X, Y, and Z must really not be so very good at X, Y, and Z at all, and must be (over)compensating for some actual deficiency; it’s much less pleasant to imagine that people who go around shoving their (alleged) superiority in our faces might really be better than us at what they do.
Unfortunately, Kruger and Dunning never actually provided any support for this type of just-world view; their studies categorically didn’t show that incompetent people are more confident or arrogant than competent people."
and to end a post about a paper people can't read with "Maybe I’m being naive here, but I think a major reason behind false memes is that checking sources sounds much harder and more intimidating than it actually is." -- that's just rubbing it in!
Angry venting against someone you think wrong is gratuitous negativity. So are generic dismissals like "This whole article is just a bag of fail."
If you think someone is wrong, please say why substantively, or say nothing.
A bit like Behind the Headlines, but about any science and by a wide range of science bloggers.
When media says "X causes cancer", he comes with a level headed blog-post explaining the data. When anti-vaxxers point to a study, he shows that they have misinterpreted it, etc.
Unfortunately, it's in Norwegian. I'm interested in reading more in this vein, so please let me know if you have good, English resources.
[1]: http://tjomlid.com/
However, I disagree with his interpretation of Dunning and Kruger's figures. There is limited data in those figures of course (3 sets with 12 points of data and 1 set with 8 points), and I don't know how big the actual sample sizes were. But in all 4 cases, it clearly and significantly shows that people with low actual ability overrate themselves and people with high actual ability underrate themselves.
> The pop-sci version of Dunning-Kruger is that, the less someone knows about a subject, the more they think they know.
I can only guess that he is interpreting that too literally. I always took the latter part as "the more they think they know [compared to what they actually do]". You could plot "perceived - actual" or "perceived / actual" instead of the figures in the paper to make it more obvious.
I read the data quite differently. I read that people don't use the full scale when rating themselves. The self rating scale goes from 60 to 90. If you correct for that, people kind of seem to do a nice job in self evaluation.
It can be explained, too. People have social pressure not to rate themselves near 100%, nor in the bottom half of the scale.
Edit: based on other comments, it sounds like maybe the 3rd and 4th graphs were identical, but the author has now corrected the error.
* a straight line on a log scale - which is misleading, since the man on the street is used to straight lines on linear scales - telling that money buys happiness
* a "log-like" graph on a linear scale, which tells you that "money buys happiness a lot at the start, and then it doesn't change things a lot" - and the graph uses a linear scale on both axis, so there is less deformation.
What am I missing ?
But there is no inflection point. 10% more money always buys you the same amount of happiness.
Nothing makes $75k the point where you have 'enough' money as opposed to $5k. 75 million is just as far ahead of 5 million (at least if it continues to be logarithmic).
10% more money is much more money when you're rich than when you're poor (Captain obvious to the rescue.)
So if you want to make me 10% more happy, that's going to cost you a lot more actual dollars / euros / yuan / whatever ; and the last 10$ are going to make me less "happier" (in absolute) than the first 10$.
Considering people don't buy stuff in percentage of their salaries, but in absolute dollars (if i'm not mistaken), isn't that a case where the absolute / relative difference is relevant ?
I'm pretty sure I'm having a stats 101-level argument, and there is something obvious that I'm missing, so please forgive and educate ;)
People trot this out and say "look, once you have a certain amount of money you don't get happier by having more!"
Why should dollars buy happiness the same way they buy gumballs? It doesn't make any sense.
The first gallon of water you have every day is pretty freaking valuable. The 1000th gallon of water you have every day is just about worthless unless you're a farmer.
In engineering we use log-log charts to look at how systems respond to a HUGE range of inputs from the very, very slow to the very, very fast. Like filters. You want to know how it looks from 1Hz to 1GHz in a useful way. This is generally done with a Bode plot which is log on both x and y. http://en.wikipedia.org/wiki/Bode_plot
If your plot covers several orders of magnitude (and income is many orders of magnitude) then the most informative way to understand it is by compressing it via logarithm. Here's a good example only on the Y since the company is growing X% per year.
http://blogs-images.forbes.com/naomirobbins/files/2012/01/li...
Which is precisely the point of the old "money does not buy happiness" adage, isn't it ?
Science that says "This 1000ths gallon of water is gonna be as good as worthless" seems to favor the point rather than oppose it.
I don't know if "happinness == gumballs" would make "sense" to anyone (I personnaly doubt it), but that's the subtext of a lot of messages in a consumer society (and we're leaving the realm of maths from the realm of politics. Stats 101 vs PolSci 101, all over again, I know, sorry.)
As for your graphs, what I was saying is precisely that, unless you've been trained to read graphs (and most people are not), you're going to be "fooled" more easily by the second one, won't you ?
Not the way I read it, really. If money doesn't buy happiness, then the graph should be completely flat right? Or at the very least a uniformly distributed scatter plot? But it doesn't really seem to look that way. While I suspect that there's some truth to the adage, it's definitely not absolute.
> Science that says "This 1000ths gallon of water is gonna be as good as worthless" seems to favor the point rather than oppose it.
That's not the point. The point is that as you have more of a thing, it takes more to move the needle. EVERYTHING has decreasing marginal utility, and money does too. But I don't think the marginal utility of money ever becomes zero, or goes negative. Do you think that? If so, please start sending me your money so that you can be happier!
Here's a great example for you. Raises are nearly always percentages right? Why not just give everyone a $100 a year raise? I mean, that $100 is supposed to be the same to everyone, isn't it? 10% more money increases your happiness approximately the same, no matter what number that 10% increase is applied to.
> As for your graphs, what I was saying is precisely that, unless you've been trained to read graphs (and most people are not), you're going to be "fooled" more easily by the second one, won't you ?
I think that if you want to understand the way a system works across many orders of magnitude the only way to do so without fooling yourself is to look at a log/log plot. If you don't, only the most extreme change on the plot shows up and all the rest gets compressed into a "straight" line by the vagaries of turning an infinitely variable thing into a finite set of pixels.
There is no such point.
There is no reason to favor any particular income level. If you use the happiness graph to make an argument that $50k is all anyone needs, I can use the same graph to show that $400 is all anyone needs.
If the 'up to a certain point' hypothesis was correct, with the graph leveling out, there would be a maximum happiness-from-money. You could say "oh well 85% of max is plenty, the ideal income is X". But the data says that's not true.
---
Humans intuitively understand logarithmic scales. Arguably even better than linear scales. A lot of our senses are logarithmic, after all. It's obvious that one dollar does not buy one happiness, given that millionaires aren't in ecstatic shock. But every time I double my income I gain the same number of happiness units? Sounds like money buying happiness to me.
Also, as another "I don't get stats get me out of here" question, isn't it problematic to plot against a "perception" level that (unless I'm mistaken) is linear ? What do you do with someone that answers "I'm happy at a 10/10 level". Do you consider it an outlier by definition ?
Assuming the "happiness" self-rating is indeed linera, is it still a fallacy to think that, behind the mathematical rightness, there is value in knowing that after $x/year, y% of the population rates itself as above, say, 8/10 ? I'm under the impression that it's a different argument than say "after $x/year, everyone is happy" ; but that it can still be practical (especially if you can corelate lower levels a happiness with stuff you want / should / need to get rid of. But again, talking politics at a math cocktail, probably bad manners.)
People rating out of 10 distorts the numbers in multiple ways, that's a separate issue to deal with. Ideally you'd remove self-reporting in some manner...
>value in knowing that after $x/year, y% of the population rates itself as above, say, 8/10
If it's linear with a cap of 10 then all the reasoning from before goes in the trash. The graph won't actually be logarithmic and you can set a cutoff easily.
If you can calculate a linear happiness score with no cap, then it's possible to pick a point that's "happy enough", but that point will be arbitrary. It won't be based on an inflection point on the graph, because the graph has no inflection points. You can use your judgement to say that 12 happiness points is plenty, but that's not a math question.
For those of you who earn the requisite amount, is it true that it's easier to get a significant raise at the higher income levels? I live in a fairly rural area but I've found as my income rises the degree to which I need to fight for marginal pay increases has diminished, which seems counter-intuitive since each point gets more and more expensive, obviously.
As for getting larger raises, many jobs you get raises based on a percentage of your current salary. So if you get a 5% year-over-year raise and your base salary is $120k, you're getting a similar happiness boost as the 5% year-over-year raise of someone making $40k base salary. But you're getting way more in terms of net actual dollars.
Each figure shows how the bottom quartile is overestimating their ability. And that's exactly what people call the Dunning-Kruger effect.
It should probably be http://danluu.com/images/dunning-kruger/dunning_4.png which shows a positive correlation for "perceived logical reasoning ability and test performance". That then helps explain the statement "In two of the four cases, there’s an obvious positive correlation between perceived skill and actual skill".
The pithy version is "a lie can run round the world before the truth has got its boots on", with thanks to Pratchett.
I think your comment contains a highly recursive component:
What Dunning-Kruger tells us is that, with regard to certain abilities and especially social skills, people are bad guessers of their own competence and that most people think of themselves as slightly above average.
I think that he's also somewhat right on money vs. happiness. Self-reported happiness is not the same thing as actual happiness. You might rate yourself a "7" at two times of life, but be behaviorally different. Your "7" when you are poor might be, "commute sucked, but no one yelled at me at work today"; but when you're rich, it might mean "hotel is nice, but staff forgot to fold the toilet paper into a triangle". Self-reported contentment is the same, but stress levels and behavioral measures of happiness are very different. It is true that after $75k per person, other factors have more of an effect on happiness than money itself. You have to account for cost of living and family size, of course.
Then there are job quality issues. A writer who reliably makes $100k per year is a smashing success; a professor earning that, at 40, is more than respectable; but someone who's a "Software Engineer II", making that, at age 40 is a failure. I think that success is much more strongly correlated to happiness than income. People with Harvard MBAs making $250k per year are often the most miserable people on earth, while famous artists who make a fraction of that often love their lives.
His relating this to type systems is a bit silly. You can't put all "static typing" languages in the same bucket. Java's static typing is a hindrance with little power. Haskell's static typing is extremely effective-- if you know how to use it. Also, quality of the engineers matters more than the language itself: I'd much rather work on a disciplined Clojure or Ruby team than a sloppy Scala team that's still using Java patterns. If it is properly used, a Haskell-like type system can make code quality very high indeed. That said, most businesses aren't willing to budget the time for quality code, and that's a language-independent problem.
This probably has to do less with the utility of the money itself and more with psychological positive reinforcement ("I'm smart and successful and my boss recognizes that") as well as increasing the person's relative social status. I've read that, given the choice, people will prefer to (for example) earn $70k while their peer group earns $50k, as opposed to earning $100k while the peer group earns $150k. Income is an indicator of social status and is largely valued relative to the rest of your social cohort.
Interestingly, the Japanese are said to be more efficient because their corporations tend to award social status directly to "salarymen" (sic) rather than indirectly through differential wages.
It's just saying that people tend to overestimate their knowledge of a subject when they are under-educated and tend to underestimate their knowledge of a subject when they are over-educated.
Of course, the study participants were asked to give their own estimates. What they were actually thinking of their abilities is a matter of debate that would need a more clever study to deduce.