Should this paper in psychological science be retracted? (2016)
statmodeling.stat.columbia.edu
statmodeling.stat.columbia.edu
If you don't want to get too fancy, a cubic polynomial will probably predict a fake huge advantage for teams with very high talent ratio.
I expect something like f(t)=C-expt(t/K) that looks like a quadratic for the low values but it's very flat near the high values.
I found many times good fits with f(t)=A log(t/B)+C.
In any case, the solution is to have a good model and amore data. Both are very complicated here, and extending the trend is a mistake, even as a unsupported conjeture.
I'm not sure if is should be retracted, but if you make a graphic like this in the physics lab, the T.A. will make you redo the whole presentation.
Also, the grow phase is quite exponential, so even for a polynomial fit I'd prefer to use it after changing the vertical axis. Otherwise the polynomial must fit too many almost zeros and then jump, and you will get weird and negative results.
There was also a strong weekly variation, cases on weekends were written down on Monday (or Tuesday), so I think it's better to look at the 7-day running average.
The problem is that in this paper they are extending the "forecast" too much. It's somewhat the equivalent of 6 months. I thin no serious source tried to make very long term forecasts. (Anyway, some trends and general forecasts are possible, like with the flu.)
The Y axis is the mean score of the team, but that score is not independent from the opposing team.
It means that it does not refute the hypothesis that sparse top talents in a team happen at low level games, where the top talents make a huge difference.
When reaching higher level games by acquiring new top talents, the playing field is more leveled, and the scores will even out.
To remove that effect, you would need to select games where a team score does not depend on the opposing team score. Or add a new dimension to the regression which is the opposing teams top talent %.
TLDR: if you put Michael Jordan in an amateur team, he will overperform and crank up the team score. If you then add 5 other pro players to the team, you'll enter the NBA. Your team will perform better, even though your number of points may decrease, as other teams are closer to your elite teams level.
The paper is garbage, but this is not true. All Y-axes are how they define "team performance" -- in the case of the FIFA study, it's qualification rankings. In the case of the NBA/MLB study it's end-of-season team win percentages.
The argument still holds though. Overall team disparity should be much higher at low level than high level.
A team composed of "Michael Jordan + random players" would crush any amateur league with 100% win rate.
The same team with "Michael Jordan + Magic Johnson + random players" would probably get 75% win rate in an NBA match.
So the metric "win rate" alone, or any other metric non independent of the opposing teams level, is naturally useless in this study.
Seems like without this meta data over the original article that were destined to keep consuming even retracted knowledge.
'I think the educational and psychological studies I mentioned are examples of what I would like to call Cargo Cult Science. In the South Seas there is a Cargo Cult of people. During the war they saw airplanes land with lots of good materials, and they want the same thing to happen now. So they’ve arranged to make things like runways, to put fires along the sides of the runways, to make a wooden hut for a man to sit in, with two wooden pieces on his head like headphones and bars of bamboo sticking out like antennas—he’s the controller—and they wait for the airplanes to land. They’re doing everything right. The form is perfect. It looks exactly the way it looked before. But it doesn’t work. No airplanes land. So I call these things Cargo Cult Science, because they follow all the apparent precepts and forms of scientific investigation, but they’re missing something essential, because the planes don’t land.'
"It’s a kind of scientific integrity, a principle of scientific thought that corresponds to a kind of utter honesty—a kind of leaning over backwards. For example, if you’re doing an experiment, you should report everything that you think might make it invalid—not only what you think is right about it: other causes that could possibly explain your results; and things you thought of that you’ve eliminated by some other experiment, and how they worked—to make sure the other fellow can tell they have been eliminated."
Using computers to parse statistical data is interesting and can reveal truths about society. Trying to correlate the delay between eating a marshmellow as a child and income as an adult is ridiculous, yet collectively we have spent untold money and time on tenured professors, conferences, papers, and pop psychology books about such things. Another example is Power Posing. How Amy Cuddy still gets employment is a mystery.
If I were the parent of a child who ate the marshmellow too quickly, I would do nothing different, as the whole thing is meaningless and doesn't change anything. It's more like, "Oh, OK, my son is 4% less likely to be in the top 20% of the income distribution (according to this barely coherent theory)." Why do we spend so much money, time, and effort discussing and researching this?
Teaching a child to do basic budgeting with pocket money is likely to have a vastly higher impact than slight delayed gratification with candy.
Further, the result in a recent replication [1] was a correlation of .28 between the time to ring the bell (to get the marshmallow) and academic achievement. That's not exactly the trivial "4% less likely" effect in your caricature.
(You may also wish to double check your spelling of "marshmallow.")
[1]: https://www.jasoncollins.blog/posts/the-marshmallow-test-hel...
You clearly don't know how to interpret correlation. A correlation of .28 is virtually uninformative. https://www.academia.edu/39797871/Fooled_by_Correlation_Comm...
I.e. it's a covert test if parents are rich.
The only thing that children of rich parents share is to have rich parents. One of the parents being over the mean in the "beautiful" trait, is also very common.
Do you have evidence to exclude other possible interpretations, such as the possibility that children resisting the temptation of a marshmallow and their parents being rich are actually the effects of some other underlying single cause?
Stop and think.
If you are rich that marshmallow is probably very bland or you've eaten very well today. So maybe two makes more sense.
If you aren't, that marshmallow is super tasty and you probably didn't eat well today. Plus do you believe that grownup when your mom was lying. Someone might come and eat it.
Then some time passes. Rich kids grow up and are more successful. Delayed gratification is key to success!
https://behavioralscientist.org/try-to-resist-misinterpretin...
Marshmallow effect is real, but has important confounders like "stable home life with trustwor5 adults".
https://statmodeling.stat.columbia.edu/2017/10/18/beyond-pow...
We'd probably all agree that's worth keeping an eye on your own credulity. That applies both to believing the scientists themselves and to believing critics that the science is overall bunk.
I believe Feynman when it comes to psychology.
"'I think the educational and psychological studies I mentioned..."
This is not a wholesale dismissal of the entire domain. That so many people interpret it that way is a psychological phenomenon in itself.
I explained to her that it was necessary first to repeat in her laboratory the experiment of the other person—to do it under condition X to see if she could also get result A—and then change to Y and see if A changed. Then she would know that the real difference was the thing she thought she had under control.
She was very delighted with this new idea, and went to her professor. And his reply was, no, you cannot do that, because the experiment has already been done and you would be wasting time. This was in about 1935 or so, and it seems to have been the general policy then to not try to repeat psychological experiments, but only to change the conditions and see what happens.'
Judging by the replication crisis in psychology, not much has changed since Feynman's day.
It has never been acceptable in psychology or any other scientific discipline not to include a control condition in an experiment. You may not know that many statistical tests were developed by psychologists, and nearly all early development of statistical testing was done by by social scientists of all kinds. Same tests still used by data scientists today.
You seriously think that because Feynman happened to advise a psychology student with a poor adviser in 1935, that an indictment of the science 70-90 years later, that was then thoroughly embraced and prioritized by the entire discipline, even though the crisis was mostly applicable to a sub-field (social psychology), still discredits that entire area of science?
You're certain of that, even though you aren't in that discipline and have no professional responsibility to maintain that expertise?
Either way, rethink your standard for certainty. This whole jack-of-all-trades but also an expert-at-all-trades nonsense has to stop.
I am not an expert in astrology either, but I am certain it is bunk.
SPE was garbage science and all attempts to replicate the results failed. Your deflection of 'ethical issues' underscores the problem with psychology when you ignore the fact that a hallmark study in the field was rigged by the experimenter to get the results he wanted.
As for textbooks:
> Zimbardo’s 1971 Stanford Prison Experiment (SPE), one of the most famous studies in psychology, is discussed in most introductory textbooks. The present study is concerned with the nature of this coverage, given that there have been myriad criticisms, especially recently, of the SPE. These criticisms concern both Zimbardo’s situationist explanation of the outcome and the study’s methodology, such as the presence of strong demand characteristics. Thirteen contemporary introductory textbooks were analyzed for their coverage of the SPE and the ensuing criticisms of it. Eleven of these texts discussed the SPE, but only six even mentioned any of the criticisms. Possible explanations for such coverage and a plan to incorporate more accurate coverage within the discussion of research methods are offered.
> ...
> Of the 11 texts, 5 did not include any criticism of the SPE, and the other 6 provided very minimal discussions of such criticism. One text included a sentence questioning the ecological validity of the SPE results but did not provide a reference. Three texts briefly discussed ethical questions created by the SPE, such as the question of whether the ethical costs of the SPE outweighed its scientific gains but did not provide references. One of these texts also stated that many researchers have challenged the legitimacy of the SPE, which would seem to be referring to the recent criticisms of the SPE, but no references were provided.
Introductory survey textbooks are essentially enrollment propaganda. They are bad places to seek expertise. Some psychologists make it part of their life work to write better ones. But it's psychologists who are criticizing and working to improve the field.
Sorry to say, I have to disengage to drive my family home.
Consider the nature of this style of thinking from a psychological and logical/epistemic perspective, the various different forms it can manifest in, etc.
Is this supposed to be a gotcha? Those have mostly been used as Ponzi schemes or hobbyist toys that extract wealth from those who don’t have it to lose
My aging mother lost her retirement savings to crypto "investing" despite my attempts to dissuade her for a few years. I do think that privacy-focused cryptocurrency is a legitimately worthwhile area of study/development even though the current industry is beyond a dumpster fire. But the people who are serious about it as an area of inquiry are going to stick around and use the mistakes of the past to improve. The people in it only to make a quick buck or billions probably won't.
As with nearly everything, there's more nuance involved than armchair critics tend to appreciate.
The paper adds to the growing body of evidence that DSM-5 diagnoses lack validity. The study specifically looks at ASD, ADHD and OCD in children/adolescents, but there are other studies similarly examining other diagnoses. I think this is very important research, because if we are going to use psychiatric diagnoses, we have to support research which questions the evidence behind them. If we don’t think that kind of research is worthwhile, then either we stop using them altogether, or else we are turning them into something faith-based rather than science-based.
In my personal experience, many clinicians – and even researchers – talk about diagnoses in a reified way which only makes sense if one presumes they do have some sort of validity. Many of them also appear to engage in the "motte-and-bailey fallacy" – talking about diagnoses as if they were valid, falling back on "we all know they aren't really" whenever that talk is challenged, and then going straight back to talking that way as soon as the challenge has departed.
If one takes the lack of validity of DSM-5 diagnoses seriously, that has certain consequences for research programming and study design – that one ought to prefer trans-diagnostic studies to those focusing on single diagnoses, that one ought to study cohorts which include subclinical cases along with those with a clinical diagnosis, etc – and yet, very many researchers continue to ignore all those recommendations, which suggests they don't actually take that lack of validity seriously at all.
In 2017, the journal Autism Research published a letter to the editors, by Lynn Waterhouse, Eric London and Christopher Gillberg, entitled "The ASD diagnosis has blocked the discovery of valid biological variation in neurodevelopmental social impairment" [0] – the authors argue that focusing research on a diagnosis which lacks validity is a scientific dead-end, that it is no wonder that it has failed to produce the results which its promoters had promised, and that the field is going to continue to be fruitless until research is re-oriented away from its current focus on an invalid diagnostic category. And I'm sure they'd say that, while their letter was about ASD specifically, the same is likely true for many other DSM-5 diagnoses as well
> Most of them are just clusters of highly correlated symptoms without a clear, objective causal link to any underlying pathology.
How strong is the evidence that these symptoms actually are "highly correlated"? To take ASD as an example, the diagnostic criteria refer to problems in two separate domains – social communication, and restricted/repetitive behaviours/interests (RRBIs) – with problems in both domains required for a positive diagnosis. This linkage is justified on the grounds that the two symptom domains are "highly correlated". However, the scientific evidence for the strength of that claimed correlation is actually rather weak and lacking – see [1]. And if that is true for ASD, it is probably true for many other diagnoses as well
> This helps to facilitate communications between providers
If the point was really to facilitate communication between providers, a better approach would be to just provide a list of symptoms for each patient/client. DSM-5 diagnoses are less useful than that, because many of them are defined in terms such as "Exhibits at least M of the following N:", so two people can both have the same diagnosis without actually having any symptoms in common.
> as well as in generating the documentation required by payers to justify insurance claims.
Well, here we get to the real point of diagnosis – a collection of cultural constructs in which policy-makers have faith, so they'll construct policies (insurance, disability service funding, research funding, drug approval, etc) oriented around those cultural constructs – and they display a studied ignorance of the lack of scientific evidence to support what they are doing, and also the very real possibility that doing things that way is at best useless and at worst positively harmful (see [2])
[0] https://pubmed.ncbi.nlm.nih.gov/28714261/
They're also 50 years old.
If that's your idea of what most of psychology is like, then I imagine you think 90% physics is all about things like phlogiston.
Just because there are some very well-known experiments that were very poorly conceived and conducted does not say anything about the field as a whole.
As someone who works in tech for a psychology department, I can tell you from a decade's experience that a) the people there care very deeply about performing good science, including replicating results, pre-registering papers, and doing good statistics (not p-hacking), b) there's a lot of psychological research that's decidedly unsexy, very much about things like proving what we already intuitively know to be true or sussing out small nuances between X and Y, and c) while they teach about Milgram and his ilk, it's almost entirely in a "What Not To Do" type of way.
If only!
Cancer biology has also struggled mightily with replicability, as has machine learning. Amgen could only replicate about half of the published findings in their reproducibility project; I’ve reviewed MLmreplications and the rate isn’t much better. Some of this is undoubtedly for petty reasons—-people run bad experiments, or put their thumb on the scale to get a particular result.
However, another major issue is that the “soft” sciences are actually quite hard. There are a million factors that might affect how an organism responds in a test. We often can’t control for all of them, and may not even know what they are. In a sense, electrons are easier to study. They’re all exactly identical, they only interact with each other in a few simple ways, and they don’t care at all about what you, the experimenter, might want or expect. Every patient (or rat) is unique and susceptible to all kinds of influences based on their interactions with each other, the experimenter, and the rest of the world. In that sense, the “replication crisis” just means we’re slowly and awkwardly discovering what matters.
It was on the softer side in Auguste Compte’s original 19th century definition, but things have moved on quite a bit since then. Platt’s famous article on “Strong Inference” explicitly calls out molecular biology as a success—-and does so ahead of physics! Storer calls biochemistry a hard science and botany a “medium hard” one.
The whole idea of a single, uni-dimensional spectrum of “hardness” is pretty silly, but even within it, the natural sciences (especially in the lab) are pretty unambiguously “hard”.
Its findings are somewhat less suspect than Zimbardo's in the Stanford Prison Experiment (which is now fairly widely panned as a godawful experimental design and implementation, even apart from its ethical problems), but it's still seen as a very big signpost saying What Not To Do.
You are right, probably. I'm not exactly sure, because "top papers" like Standford Prison Experiment, are at the top for reasons that seems to me arbitrary and far away from a science. There are a lot of very good articles, some authors are really dig into their problems. Though these are not known for a public. They do not make good headlines. It is really annoying, because basically all popular psychology is a junk-science or have nothing to do with a science at all.
But in any case we can agree that there are a lot of papers with little to no value. And as I see it is because psychology and sociology are on the quest of creating the Method. The Method of research that works in their discipline. They do better now, then it was before. Now a paper like Zimbardo's wouldn't stand a chance. Papers from 30 years ago probably wouldn't seem rigorous enough now. The Method is becoming better, and probably it will become even better in a future and today's research will be deemed not good enough to be science, and will be read by undergraduates as a cautionary tale how one mustn't do Science.
It is a little like programming which changes with each passing decade, though a little slower, I'd guess. But socially it is different. I mean if we look not at results of research but on a social processes leading to this result, it is different from programming. Programmers are expected to create programs that work, they need to deliver. There is a final clear criterion of a success. Psychologists do not have such a criterion. They seek Truth, but what Truth is and what isn't is debatable. And they work for grants, not for revenue. Programmers are inventing new tools and techniques allowing to create better or more complex programs, but at the same time they also need to deliver something useful for non-programmers. Psychologists are inventing new tools and techniques of research (or sharpen old ones), but they do not feel the urge to deliver something to non-researchers. And to be honest oftentimes they are unable to deliver. They can find some grain of truth, but no one can find a practical application for this grain yet.
I think it will get better when neuroscience, AI, and psychology converge and some kind of a psychological theory will emerge. They are doing it now, and they are bound to collide at some point in a future. There are lots of psychology theories now, but no physicist will agree that any of them deserves to be called a theory. They are more like vague descriptions than theories. Or they are talk about some small part of mind.
I cannot say about prospects of sociology because I know little about it, but I'd say it would take more time to bring sociology to a high standards of an empirical science.
> Trying to correlate the delay between eating a marshmellow as a child and income as an adult is ridiculous, yet collectively we have spent untold money and time on tenured professors, conferences, papers, and pop psychology books about such things. Another example is Power Posing. How Amy Cuddy still gets employment is a mystery.
May be it is ridiculous. But if we want to know how human mind works, we better pay for it. Researchers played with electricity for centuries. Michael Faraday himself spent half a century designing experiments. All that didn't deliver nothing of a real practical value. But then James Clerk Maxwell published his equations. And suddenly electricity started to deliver. Electricity delivered probably more than any other technology.
Oh no they definitely have value, in the same way that metaphysics has value
It includes tons of work characterizing sensory perception. These studies tend to replicate pretty well and they have a pretty broad range of real-world applications: psychoacoustic models for audio compression, principles like pop-out and serial/parallel search that inform user interfaces, color reproduction. In fact, psychophysical experiments pinned down the properties of cone cells over a hundred years before they were directly measured.
Psychometrics, the study of test/assessment design, also shows up all over the place: education, obviously, but also marketing and other forms of evaluation.