Ego depletion theory may have been debunked
slate.com
slate.com
> "For a second paper published last year, Carter and McCullough completed a second meta-analysis that included different studies, including 48 experiments that had never been published. Again, they found “very little evidence” of a real effect."
> "“All of a sudden it felt like everything was crumbling,” says Carter, now 31 years old and not yet in a tenure-track position. “I basically lost my compass. Normally I could say, all right there have been 100 published studies on this, so I can feel good about it, I can feel confident. And then that just went away.”"
So you have at least 100 bullshit studies on a really popular and often citied experiment. If bullshit can hide in plain sight like this, imagine what is lurking out of sight?
There seems to be a few issues at play. First is the "file drawer effect". If the experiment doesn't show a positive (or surprisingly negative) outcome, then it's most likely to get filed into a drawer and forgotten.
The second issue is just random luck. Suppose you flip a coin 10 times, then it's possible that you might get a run of nine heads and one tail. Based on that experiment you could publish a paper that the coin has a 90% chance of flipping heads. That would be a surprising result, and it's likely that it would be debunked if anyone tried to reproduce it.
The third issue is that it's exceedingly rare for anyone to try reproducing an experiment. There's no grant money for reproducing experiments and regardless of the outcome it's unlikely to advance a researcher's career.
Together these mean that most published papers could very well be the product of random luck that will never be verified.
For instance, in the first experiment outlined they had one set of subjects eat radishes instead of cookies before trying to solve an impossible puzzle. Now, if you are a subject and the first thing you are asked to do is eat radishes, isn't this a pretty good signal that you are probably being messed with? The logical reaction to that is to conclude that there's a greater chance that the puzzle is also some sort of trickery and you are much less likely to spend a long time trying to solve it.
The mix of students in PSY 101 were people like me stuck with taking it and people who wanted to enter the field. I imagine both populations skewed the results.
The practice of science is a very human affair. That means politics and clan thinking. Science, like the press, develops a popular "narrative" about how things work. New research tends to magically confirm that narrative.
I suspect this has been going on for a very long time.
I love science. And I understand that there are a lot of demagogues that want to create these false battles between science and the ignorant. For many of those folks, they will want to circle the wagons. Don't let folks know how bad things are!
To those folks, I'd ask them to remember that the strength of science is its tentative nature and ability to adapt. Sure, people aren't so good at that, but science is. Stick to that and do your best to toss emotions out of it.
Experiments in areas with huge numbers of degrees of freedom are difficult to sort out.
I mean, if we just take just the chocolate cookie experiment, and assume that we can replicate that particular experiment. We run 100 versions of that experiment and find that students will attempt the impossible for about twice as long as those who had to eat radishes.
What is the most general thing that we can conclude?
How about if we find the same result with candy vs celery? Twinkies vs nothing?
What can we possibly be sure we learned from such experiments?
The question then would be who would want to provide funds for that? It is not the financial backers who have problems with the current methods.
I don't think the butter vs margarine or the tea vs coffee debate will be settled within my lifetime ;)
Science is a lot like sausage processing. The end result may range from the quite passable to the quite good, but if you actually saw the raw materials and the process, you'd be more discriminant on what you put in your plate.
If I publish something claiming bacteria has a gene X that produces phenotype Y, then a subsequent study that transfers gene X into a different bacteria must also observe Y. Otherwise something is fishy, and often the second set of researchers will go back and reexamine the original result.
It's rare that work in science is directly, point for point replicated, it's unsexy, expensive work. The fact that the harder sciences have somewhat of a built in control mitigates but by no means eliminates the risk of bad science surviving.
I can't think of an analogous process in the softer sciences. Seeing if a phenomenon is replicated in a different context does not implicitly re-test the original work in the same way.
The problem with results like "ego depletion" is that their utility is fuzzy at best. With no clear application, we don't have a body of practical results using it.
There is no guarantee (and it often isn't the case) that the phenotype observed in the first organism is purely the result of gene X; its effects may be context-dependent.
To take this a little further, I think it's plausible that a soft scientific question differs from a hard one specifically in terms of how easy it is to separate the phenomenon in question from its context.
The reality is that the end results are usually terrible and a lot of science out there is straight up junk science designed to chase grants and to lean on journal biases to get published, especially in the social sciences. The problem is we don't really have a better alternative so working with a clock that's right twice a day is much better than having no clocks at all. We might have hit organizational limits on how well we can do science on this level. With funding comes politics, with journals comes bias, with publishing comes perverse incentives, challenging big egos is difficult, unknowns have to 'pay their dues', small universities and institutions don't have influence compared to their larger peers, the sometimes political tenure process, etc.
Hell, the article itself even calls out Carter's relatively young age or for not being tenured in an almost "Can we trust this young nobody," way.
That's normal science at work. Old sages are biased, like everybody else. Fortunately, they are mortal too [1], and science keeps moving forward 'one funeral at a time', as the saying goes.
The problem is with people who want to substitute faith in the Divine Providence of God, with faith in the (eventual) Unfailibility of Science. We are just a bunch of social primates, after all. We were never meant to peek into the Deep Mysteries of Existence, anyways... which does not mean we cannot use our relatively advanced intelects to figure out a bunch of clever and useful tricks.
======= edit: comment [1]
[1] Something that is nightmare fuel to me is that one day we are going to stumble upon some serious life extending technology, and then we will all end up under the thumb of a bunch of ancient, obstinated fools who cannot/won't accept the limits of their own leadership skills.
But this is a very large literature, that in effect by sheer volume of work was doing many of those things. This is quite humbling.
That said, I do think that medical science, though massively flawed (see previous discussion on clinical trials no longer revealing positive results https://news.ycombinator.com/item?id=11209778) there is a way to make it so we can trust science. By:
1.) as previously said given a hypothesis, it must be confirmed by 2 independent approaches (if this is true, then this other thing should be true, is it?)
2.) publishing more negative results, removing the incentive for the file-drawer effect mentioned by @jobu
3.) require much larger sample sizes for studies to be published, and take effect size properly into account when judging whether the outcome is valid. One interesting way to do this:
http://www.stat.columbia.edu/~gelman/research/published/retr...
That a team claims that there is a problem it is a good sign that this needs more studies. But it will be jumping to conclusions telling people that the original study has been debunked.
However, consider if the two studies were reversed in time, so the first study specified what its endpoints were, then did the study, and concluded there was no impact.
Then the second study came along, and during data analysis an interesting correlation appeared which showed there was a impact.
Assuming that both were done by equally competent researchers, which conclusions would you trust more, and why?
Do your conclusions change if you swap the time order?
Personally, I would trust the study with the well-defined endpoints over the other one, but I also feel within myself the desire to trust the first one - even though the order shouldn't change things.
I think it means ego depletion is much more fragile than we thought or is caused by some other phenomenon. I don't think this study means the thing we called ego depletion doesn't exist and is an artifact of publication bias and chance. It seems to me like there were just too many studies replicating it for that to be true.
Is this an airtight argument? No. There is still the possibility that global warming research has analogous methodological issues. But the conflict of interest definitely raises the burden of proof.
An additional difference is that climatology research is much better founded than willpower research is. Psychology is notoriously tricky to test since you can't break apart a brain and watch all the parts interacting. Compared to psychology, the systems that climatologists study are downright simple, and we have good models of how the systems behave.
For climate deniers, the evidence just isn't there. There's no wave to ride.
Psychology is among the social sciences. That field is experiencing the reproducibility crisis, and it's been a long time coming. The subject matter is extremely difficult to study in an objective way, and the smallest of nuances can distort results (e.g. the article mentions taking a test on computer vs on paper). Also, humans aren't passive subjects, they change and react to being studied.
That's just not the case in the physical sciences like chemistry, physics, etc. There isn't any similar crisis there at all.
To be fair, climate science is somewhere in the middle, in that it can't be studied in the fully rigorous way that physics can, with controlled experiments. However, that middle ground contains other very solid fields of science, like evolutionary biology. Massive reproducibility crises just haven't happened in those areas.
The social sciences are just different. Perhaps it's naive to think that a simple law like "ego depletion" could possibly work consistently, given how complex human beings are. That's the core issue, and climate science doesn't suffer from it.
So for example, in the cookie/radish story, were all participants exposed to the cookie smell? Were they aware some people got cookies and some people got radishes?
If so, the reduced performance of the radish group could be related to a feeling of being pushed down in the social hierarchy, a self-esteem blow, because they were denied a perk other participants received.
I intuitively feel some of these psychological experiments suffer from this. That there is another phenomenon which is really close to the one they think they are studying, which is how they get it wrong. Because the brain/sense of self are both such complex interrelated systems and it is extremely hard to make distinctions between different elements of the process.
Isn't it worse than that? A statistical hypothesis test will essentially just let you refute the null hypothesis, it provides no confirmation at all for a specific alternative hypothesis (unless of course it's the only possible alternative hypothesis).
A funny story comes to mind: I studied physics at university but did one semester of psychology. One day in class we did an experiment that was supposed to show that memory is organized into something called schemas, but the experiment seemed severely flawed to me and I couldn't help protest that a much more reasonable interpretation of the outcome was that "people who have actually been told a story will be better at retelling the story, at least when compared to people who have not been told the story." On hearing this criticism my professor became silent for a moment and then said something along the lines of:
"I have a colleague who's son borrows his bike when his own is broken. This colleague sometimes says that his son thinks that his bike will mend itself if left to its own devices. Well, that's one way to interpret the data..."
I thought it was a brilliant way to acknowledge my criticism. But my fellow students unfortunately didn't seem to get it. I guess psychology was never for me. :)
There is only a few special case exceptions where this presents a problem, which are mostly thought experiments in astronomy where the evidence will be lost in the future.
I doubt the actual models get used outside of academia and central banks. The frameworks I have studied have been so situation-specific and strong-assumption-driven that it would take much more time and effort to cram reality into a model than to simply use high level concepts, some basic data, a few customer interviews, and intuition to inform strategy.
Of course, most people don't actually even see what the experts say, they just take on the belief of what one person reporting on one expert says... so you get misconceptions like http://danluu.com/dunning-kruger/
I'd prefer if leaders focused on being good leaders and not trying to play expert or even knowledgeable-enough on technical matters that really don't concern them. A good leader will find the domain experts and drill them for critical information needed for their decision making, they don't need to understand the domain details. (And they can verify their trust with other experts.)
i don't know if i've ever gained real insight into human behavior from reading the results of a contrived experiment.
Therein lies my problem. :-) I want to know more, but most sources are woefully unreliable.
Baumeister has yet to be steamrolled by the reproducibility crusaders
If only they had used green jelly beans instead [^1]...
This is not limited to social psychology. Scientific publishing has a strong bias towards positive results. As long as null results are filed away, this kind of thing will continue to happen.
If your proposed effect is real then you should have no problem in proving it. If it doesn't exist, or if it does, but is much smaller and doesn't apply nearly as broadly as you have stated, then of course you will have "not much fun" proving it. Especially if that fun means making tens or even hundreds of experiments, then exhaustively processing data to find some experiments with positive result and discarding the rest, then again looking for explanations why these positive result experiments are right, and null or negative are somehow flawed. Not only that, but you have to avoid contradictions with older studies.
At least, up until whatever-it-is becomes established enough that tearing it down is novel and surprising.
Once everything that was tearable is teared down, the remaining picture will be much more reliable representation of what is real and what is not.
Science is a human endeavor and as such is subject to lots of economic and mass psychological effects, but at least there is some constraint on what can be claimed. The truth is out there, in physics and other hard sciences. In everything else, the models are much more handwavy approximations of underlying complexity and variation.
Hmm...what was different this time?
Psychology is an interesting field, because one really has to wonder just how static human beings actually are over time...It seems almost probable to assume people living today would handle the same experimental setup quite differently than those who lived 50 years ago.
Are there any widespread pedagogic or management practices that are built on top of ego depletion?
See: http://www.fastcompany.com/3026265/work-smart/always-wear-th...
Science is a process. It's a method where you conduct a series of disciplined, reproducible experiments and use this data to build a model of the truth. Science never gives you the truth directly; instead, you get a series of successively more accurate representations of the truth that you can use to make predictions. Those predictions are not guaranteed to always yield a positive result, but when they come up negative, you have more data with which to refine your theories.
Casting this in black and white terms - "this effect has been debunked" - is doing a disservice to science.
Rather, a negative result in this study should be grounds for further investigation. Why did it come up with different results? Is there something different about how they ran the study? Is there something different about the subjects, something that may hold useful lessons in its own right? Was the original data flawed? Does the effect hold only under certain conditions? All of these would be very useful things to know. The point of science is not to say "This is true and this is not", it's to be able to create increasingly refined models of "This is true under these conditions, and if you apply this model in this situation, you will get these results."
By analogy with physics: the ancient Egyptians (the Ptolemaic kings) believed that the sun revolved around the earth. To make this model work, they had to invent increasingly more complex epicycles in the motions of planets. Copernicus came around and showed that the earth revolved around the sun in a circle. Kepler re-examined this data and said no, planets actually move in ellipsis. Einstein re-examined this data and came up with a tensor calculus that even more accurately predicted the orbits of the outer planets. Now, there is strong evidence that Einstein's theory of general relativity is "wrong" too, but we don't have anything to replace it with.
Was the Ptolemaic calculation wrong? Well, technically not - as my physics professor pointed out, they were just doing a Fourier decomposition of the observed data. And for a kid making a sun-dial on a beach or a hiker who wants to know when the sun will set, the first term of that series is perfectly adequate. But the advantages of the heliocentric and relativistic models are that they can predict the data with increasing accuracy, without adding extraneous magic numbers.
Similarly, when a psychological study can't be replicated, it doesn't necessarily mean the basic idea is useless. Rather, it means that we should look further into the situations where that idea is useless, and refine our models so they gain additional predictive power.
I think that in the - day-to-day usage of the word / most commonly understood meaning of the word - 'true', it is fair to say that Science is in the business of determining truth from fantasy. Sure, you are permitted under science to investigate anything and everything that pleases you/(your donors :) ). I think that scientific consensus is very closely associated with 'truth'.
I totally get the larger point you're making, but even the nature of science itself is a bit of a grey area. I think that some ideas should, and have been laughed out of the room. The so called man-on-the-street does not have time for subtlety or scientific hypotheses which are carefully couched in nuanced terminology. I think that its OK for people to think that science has all the answers, every once in a while :)
I think that the belief that science = received truth is behind the rejection of science by a large portion of the population, because it puts it in conflict with other sources of "received truth". People discount evolution as "just a theory" because they're looking for absolute truth; if they were habitually trained to look for evidence and then knew the body of evidence behind evolution, and could compare it with the body of evidence behind gospel, they might understand that "theory" is a lot more rigorously proven than "truth".
"Science does not give us absolute and final certainty.
It only gives us assurance within the limits of our mental
abilities and the prevailing state of scientific thought."This is just so much BS. I could bathe in radish. All the chocolate cookies in the world can go hang themselves if there's radish around.
Maybe my ego feels so depleted because I just haven't had any radish yet this year. But I did have mountains of cookies.
This is a really good case of science at work. It's an experiment that specifically targeted ego depletion and couldn't find it.
I find it a bit worrying that so many people immediately agree to the conclusion proposed by these experiments. Ego depletion is one thing, but there are other well-known effects like Pavlov's Dog.
The narrative of these people is that there are bad people out there doing unreproducible work. They know they're bad, but they're just motivated by grants, low p-values, etc, and eventually they just snap and do bad science all over the place. It's an incentive problem.
So, to prove this, they... go get grants, do a meta-analysis with some nice statistics, and get subjected to the same incentive system. Except this time, they don't even have to do anything original. They just have to badly copy something else.
There's always something like this in the article:
>In his lab, Baumeister told me, the letter e task would have been handled differently. First, he’d train his subjects to pick out all the words containing e, until that became an ingrained habit. Only then would he add the second rule, about ignoring words with e’s and nearby vowels. That version of the task requires much more self-control, he says.
I really hate getting science news from sources like Slate, because you have to decompile the journalese into what the scientist actually said, but the technical terms here are that Baumeister had participants perform a fixed-target character recognition task. This is an incredibly old cognitive task that was used in some of the first cognitive psychology experiments like [http://psych.indiana.edu/tradition/Shiffrin_and_Schneider_19...] that showed that over time with practice, a controlled or manual cognitive process, like "determine if this word contains the letter 'e'" will become an automatic process.
My cognitive psychology professor was one of Shiffrin's students, so he was actually in that experiment, and was very annoyed by it, because in the fixed-target condition, you pretty quickly start noticing your target letter all the time. It'll just "pop" out at you when you're trying to read.
Obviously the exercise here won't cause that sort of behavior, but once you learn a rule like that and start having to react in miliseconds to the presence or absence of an e, even a little bit of training will create a much stronger effect than no training at all.
This is a huge study, and it would have been drastically more expensive if this facet had been included. But it also might have reproduced Baumeister's work much more fully and the answer would have been useful. As it is, this is literally more bad science, for exactly the same reason as the original science may or may not have been bad.
* throw the article silently away
or
* substitute all occurrences with "may or may not have been", which is identical in meaning. I am sure more readers would notice, that they are being fed information tripe.
Each individual's physiological composition makes psychology nothing more than a practice bed for bored creative writers. Understanding the psyche is critically important to write convincing characters, absolutely, and that takes a lot of practice and thought to fashion into a functional perspective (or literary craft tool) that enables a person to write "believable" characters. Observation, not causation, as it were.
It's not really a stretch to share this belief, if one were to be familiar with the origins of psychology that trace back to an esteemed writer and philosopher(and turn-of-the-century physician...) by the name of William James, brother of noteworty Henry James.
Some psychology experiments are undoubtedly crap, but there are also a lot of carefully-designed and well-analyzed experiments, many of which actually are casual.
Consider the vast amount of work that has gone into characterizing the visual or auditory system. In a typical experiment, subjects are shown a set of stimuli and asked to make judgements about them (e.g., "Is A brighter than B?" or "Have you seen C before?"). These results are then used to investigate the factors that affect these judgements. These experiments are definitely casual (the experimenter controls the stimulus). Empirically, these results do generalize across subjects and subpopulations and, in many cases, can be tied back to physical properties of the brain. For example, we knew that there were three color sensitive photoreceptors from behavioral data, long before the S, M, and L cones were actually identified.
tl;dr: It hasn't been 1920 for a long time.
Look at the overwhelmingly rubbish depiction of psychology in movies.
It's all people lying on couches; or tenacious psychologists breaking through with unwilling clients; or catharsis.
And it's odd, because when programmers see programming in movies they see how bad that depiction is, so I'm not sure why the meme of "psychology is just sitting around nattering about feelings" is so strong.
The Gell-Mann Amnesia effect probably best explains this. I think you might find it interesting:
> Briefly stated, the Gell-Mann Amnesia effect is as follows. You open the newspaper to an article on some subject you know well. In Murray’s case, physics. In mine, show business. You read the article and see the journalist has absolutely no understanding of either the facts or the issues. Often, the article is so wrong it actually presents the story backward—reversing cause and effect. I call these the “wet streets cause rain” stories. Paper’s full of them.
> In any case, you read with exasperation or amusement the multiple errors in a story, and then turn the page to national or international affairs, and read as if the rest of the newspaper was somehow more accurate about Palestine than the baloney you just read. You turn the page, and forget what you know.”
> — Michael Crichton
Also I promise it is a coincidence I've responded to you twice in such rapid succession!
But even within therapeutic psychology, you're right. Freudian psychoanalysis (talk therapy) was huge in literature and movies, even though Freud himself has been out of influence inside the field for decades. CBT, exposure therapy, etc are much more common treatments.
You'd be disturbed to find out just how weak some of that evidence is, especially since it's used to deprive people of their liberty and force them to take treatment against their will. There's a slew of medicine that has pretty weak evidence base. (eg knee arthroscopy)
All the same problems happen in biological sciences, and are even worse in some cases, since there's more money at stake. Failures to replicate are rampant in medical/pharmaceutical studies.
If you want to claim biology's not scientific either, go ahead. But science is about seeking knowledge, not having total agreement all the time.
Then, all of the original problems (expensive, ethical review board &ct.) still exist for those who might want to reproduce the effects, along with the fact that there is not only no incentive for researchers to reproduce an experiment, there are active disincentives.
Haven't you ever wondered why all of these articles by psychologists have been questing the very fundamentals of the way science is practiced? Psychology, as a field, is our day's premiere example of a science that examines itself. The issues with reproducibility are being illuminated and confronted by - who else? - psychologists.
You're wrong on virtually every count. The scientific method is the primary philosophy practiced by psychologists. I would encourage you to read more before making easily falsified statements like "psychology is ... fundamentally outside the scope of the scientific method."
- The classical example of a randomized experiment is a perception/psychophysics experiment: the Lady Tasting Tea. This introduced the idea of a null hypothesis as well as Fisher's exact test. https://en.wikipedia.org/wiki/Lady_tasting_tea
- Visual and auditory experiments more or less invented signal detection theory and used it to probe visual and auditory processing. Those ROC curves everyone loves predate "machine learning" by decades.
- A lot of the fancy hierarchical and Bayesian modeling work have come from psychologists and social scientists (e.g., Andrew Gelman)
http://blogs.scientificamerican.com/cross-check/are-antidepr...
Psychiatry is a really, really hard field for canonical drug research. Mechanisms of disease are for the most part completely unknown, the mechanism of action for the drugs is often unclear in the patient (why stimulants treat ADHD? it's complicated...) and you've got the additional complication that diagnosis for the disease is purely symptomatic (in fact, that diagnosis process is nearly identical to that of psychology) so two people with the same symptoms may have radically different diseases, but hey, they'll both be in your clinical trial!
Drawing a difference between Psychology and Psychiatry doesn't really hold up. Doing research on people is extremely difficult to get right, all medical and psychology research is subject to these problems.
Proper use goes like this:
1) Identify some area where theory does not match experiment or other theories (in Einstein's case, there were known experimental issues involving the speed of light that were the basis for his work on special relativity, for example).
2) Use thought experiments, philosophical arguments, divine inspiration, inspired guesses, random guesses, etc to produce a theory that accounts for your known experimental data. I expect all of these have been used in the past, in fact. Einstein used thought experiments and philosophical arguments for special relativity but an inspired guess for the photoelectric effect.
3) Use your new theory to make new predictions about experiments that have not yet been performed. If there are no such predictions to be made from the theory that step 2 produced, then the utility of that theory is pretty limited, obviously.
4) Perform the experiments whose results you predicted in step 3 and see whether you predicted them correctly. It's important to publish the results of these experiments, especially if the theory did NOT predict the results you got.
5) Keep repeating steps 3 and 4 as much as possible, ideally with a variety of different predictions produced in step 3, to gain more confidence in the new theory.
These are, of course, all "doing science". And insofar as they can be done by different people, one could spend all one's time on step 2 and be "doing science". But the scientific community as a whole is only "doing science" if it actually goes through all five steps. If it stops after step 2, that's not called "doing science".
I can tell you why: Because if it's solid science, you don't need 100 papers to describe an effect. Besides, it is a proven fact that the results of meta-analysis are weaker due to how the underlying statistics works (at least if you do it in a naive way, and I do not know if it can be fixed by altering the procedure. I have to brush up on my statistics knowledge, it's good for machine learning.).
* EDIT: I should have added here 'to investigate the ground-truth of underlying papers.' Of course there's nothing wrong with meta-analysis to investigate researcher behaviour and methodology.
And amusingly, the distribution of result is close to 50/50
A meta-analysis in those fields is usually only used to analyze the behaviour and methodology of researchers, not to investigate the ground truth of the papers.