When the Revolution Came for Amy Cuddy
nytimes.com
nytimes.com
(Also, as a side note, I'd fully expect that CS is at least as vulnerable to the same thing. McSherry et al.'s COST work is a good example of something that seems far worse than p-hacking: instead of having a clearly defined hypothesis and publishing the 1-in-20 papers that falsely state it's true, as in other fields, there's nothing clearly defined in scalable systems and 20 out of 20 people who build something excessively complicated can get a paper out of it.)
http://www.frankmcsherry.org/assets/COST.pdf
the paper you referenced?
Literally the papers are "I built a thing, and it gets faster when you do this thing to it." There's no condition on whether the base case is realistic.
To be clear, I don't think the authors of the papers he's taking down are bad people, or maliciously trying to get publications for uninteresting work, or too stupid to implement things efficiently on a single core, or anything. I think they're genuinely interested in the work they're doing, but responding to the pressures of the environment that allows them to keep doing that work.
Or you could get a large sample size to bring the p down to 0.01 or whatever. You know, do science.
It's entirely reasonable, IMO, to do more "exploratory" studies at p = 0.05, and then do a follow-up on promising studies at p = 0.01
The problem is that a study that's significant at p = 0.05 but not p = 0.01 is worth something to science. It indicates something worth doing several more studies on, preferably by different teams from different institutes with non-overlapping confounding factors. But the practice of science gives you only two options once you find such a study, as a young scientist: promote it immediately and become visible for having discovered something, or aggressively ignore it and go find something else that you think is likely to be immediately promising. "Doing science," as you put it, is the worst option for you because it ensures you spend a lot more time on something for no reward -- and the rational self-interest here is perfectly rational, because if you don't get funding or tenure, you're not going to do any science any more at all. (As seems to have happened with Dr. Cuddy.)
To be clear, I have no well-formed opinion on whether Dr. Cuddy is a good scientist, having only read this article. I assume she is, or at least that she's honest in her desires. But I think talented, honest people are nonetheless pressured by incentives, and fixing incentives is better for society (assuming we as a larger society wish our scientists to continue to do science) than assuming that people will be self-sacrificial and altruistic in pursuit of a tiny chance at accomplishing the greater good.
Meanwhile, there are enclaves where biologists are slaving away capturing data and not even realizing it's data. "Bookkeeping data" they might call it.
There should be mixers between the biologists and the computer scientists. Especially non-academic physicians and computer scientists. Look at Kaiser-Permanente, Sutter Health, or Defense Health Agency. Huge orgs, very little research. Data falling off the shelves and they don't know what to do with it.
1. A legal framework for health-related products & services that meaningfully protects individual privacy, and yet also preserves the conditions for the kind of velocity and rate of change that brings technological progress to the masses reliably and quickly.
2. Massive academic funding for more “pure” research in hardware & software design, engineering, the philosophy of what’s possible, the phenomenon and ethics of what we choose to build, and so on. TLDR Pay more people more money to do important research in the public domain, which I can’t help but think would attract many of the CS types you mentioned.
That exists, but there's upfront educational costs that need to be handled first.
> Massive academic funding
Look at Europe: their acedemics work much more closely with industry. Get a someone who can code in a room with a physician, and I garuntee they can find a problem to work on. Show a successful prototype, and we're way below "minimum viable product" here. Show a graph with real data and money will start looking for you.
Computer scientists, mathematicians, physicists, and engineers need to go to biology conferences. And biologists should be going to math, ML, and software conferences.
I still don't understand this one. The null model is (almost) always false. In this situation a small p-value is only a matter of sample size and measurement precision. How can it possibly indicate you should work more on something?
One thing I think may make sense is to get a bunch of p-values and then choose the lowest 10 or whatever to focus on for further work. There is no reason to include an arbitrary cut-off here though.
Also noticing this now... No, this is completely untrue. The p-value is not an error rate.
There will never be an objective measure for conclusion 'correctness'.
It will always be the case that you actually have to think about the data and statistics critically and decide whether the conclusions are appropriate. In certain well-defined situations, you can start to use harder metrics, but that's rarely where the controversy is in the first place.
So if you want to, you know, do science, I urge you to use that big beautiful brain of yours. As a bonus, you don't even need to commit mass rat genocide to do it.
There were (are?) enough degrees of freedom in the field in every study to always meet this threshold.
We say this, but I haven't heard any particularly good ideas on how to build such a system that isn't also easy to game. How do you ensure the failed studies were actually worth doing? Is peer review of study design feasible?
This is roughly how private industry works. I'm of course asked in job interviews about my successes, but the primary thing that determines whether I get a job, how much I get paid, whether I get other people to work with, etc. isn't what my past successes are, it's my ability to convince my peers that I am a steady and good worker. If I maintained some tech stack for two years, my employer is happy to let me continue doing that, and other employers are happy to hire me to maintain similar tech stacks, without me having been like the one person in the world to figure something out. (On rare occasion, I am, in the sense that I'll fix something upstream that's been challenging, and employers do value that! But there's no, e.g., "fix upstream bugs or perish" culture.)
As in private industry, it's gameable because it allows people doing poor-quality work to vouch for other people doing poor-quality work. But it's hard to game this long-term, because people interested in doing good-quality work will tend to find others who do, and in aggregate -- from a large number of such people working together over a period of time, not from any one individual -- these people will produce more measurable results.
Note that by "organizations" I don't mean "universities" or "institutions". Departments should probably be treated separately, as different departments within an institution can easily drift in terms of quality. Meanwhile, it should be possible recognize collaborations of researchers across researchers as an entity: if the collaboration wants to support a researcher moving between institutions and wants to keep supporting them, that doesn't lead to the same flaws, I think.
Are you aware of Registered Reports? https://cos.io/rr/
Also, if you haven't been following the 'X-ray Carrots' fiasco, that is a BIG deal in science and Science. VERY short summery: Andrew Wansink at Cornell made up a bunch of data and results, he published them, he got caught. One of these papers was about calling carrots as 'x-ray carrots' makes kids eat more of them, hence the name of the fiasco. Here's a good way into the mess [1]. Anyway, some papers were retracted, there was an investigation, yadda yadda. Now, in 'normal' science, he would have been banished from the field for this nonsense. But in 'finding-or-famine', he got off essentially scot-free. Nothing happened, really. He's still the head of the department (AFAIK), he still brings in cash, the FDA still is using his findings. This is a huge blow to science and a huge boon to Science.
The existential crisis is now out in the open: you can cheat with impunity, publish whatever, if you get caught nothing will happen, just bring in the dollars.
Now is not a good time to go to grad school. I think they're just going to have to throw out ALL US based bio-science (at least) from about 2003 to whenever heads start rolling. It's all crap, maybe, who knows.
[0] Science and science are different, though look similar. Big Science is the grant machine and the big money side of science. Splashy, barely replicated, large clean buildings, pensions, perfectly fit for corporate needs. Think Big Tobacco, BPA plastics stuff, power poses like Cuddy's, that kinda stuff. It's like the difference between the Military Industrial Complex and actually kicking down doors in Mosul. 'Little s' science is what you are probably thinking of when you think of science generally.
[1]http://andrewgelman.com/2017/04/06/dear-cornell-university-p...
I think you missed a decimal point or two here -- $42,000/month for a 100 square foot office can't be right :).
In my experience, about a third or half of the 'real' scientists I've met are firmly in "Big S" science. It's really not that different. People that complain about this a lot are often jealous, in my experience. The failures are certainly high-profile, though.
As for reproducibility, it's all case by case. We've (real? scientists) been having the same discussions about data quality forever. It just seems to have found a fresh audience. There's a reason a curmudgeonly PI might insist in Western Blots and - gasp! - Northerns from time to time. Most fields were built off of some real science. Molecular biology certainly was, and to think that just stopped happening in the early 2000s is the height of ignorance.
There is cause for alarm. I give pretty much all 'genomic' work a very wide berth unless the lab has a real genetics background, and aside for a few labs, anything with 'microbiome' in it is certifiable garbage. And clearly some entire fields are on shaky ground, namely psychology. But lets not circlejerk our pure 'little s' science ego's too vigorously.
You misunderstand, perhaps. I mean that the data itself is fraudulent, let alone the interpretations or the controls on the experiment. Many PIs have been making up the data, essentially. Pulled from thin air with a little rand() thrown on top. Now the flood gates to this behavior are open and 'accepted', at least by the people that cut the checks.
I don't think Cuddy, or anyone else, deserves to be dragged through the mud because their idea turns out not to hold up. That happens all the time, and it's not a bad thing at all.
You start to deserve that dragging more when you become too attached to your pet theory and cease to be able to drop it in the face of overwhelming evidence to the contrary.
You really deserve it when you use what political capital you've acquired to stomp on your critics despite the fact that they're actually right and you're actually wrong.
What's interesting here is that a lot of the Sayre's Law-type political machinations are taking place on a much broader stage than normal. If someone was to go after my work with this kind of exposure, you bet I'd be defensive too. It's entirely possible to have a fair and neutral conversation about research practices without shanking someone's reputation because it seems like the field needs a sacrificial lamb this week. Nobody wins, and certainly no good research is advanced, when the knives come out like this with no evidence of actual malfeasance. (And, frankly, even when there is evidence of malfeasance. like with the STAP controversy, which lead to at least one actual death.)
http://andrewgelman.com/2017/10/18/beyond-power-pose-using-r...
I thought that Gelman wasn't presented in the best of light, especially his awkward quotes about not wanting to engage in "interpersonal conflict". But he called the piece "excellent and fair to all sides" and was pretty gracious in response to Cuddy's criticisms and how the NYT portrayed him. I honestly wasn't a huge fan of the article but re-reading it after a bit of sleep, and knowing that Gelman thought it was strong, I appreciated the issues it brought up. Certainly has to be one of the longest stories about p-hacking/p-curves in mainstream media.
I sympathize with Cuddy and have no doubt that among the legit attention that Gelman and other critics have directed toward her, she's faced hatemail sparked by dumb sexism, which makes it easier to tar the legitimate criticism as being sexist.
I do think Cuddy needs to face legitimate criticism. She may not have been a malicious fraudster, but ignoring criticism after a long time is not much different than actively being deceptive. I also can believe that she faces hostile criticism that isn't equally distributed among all the other researchers who used weak, but previously acceptable standards. And I don't think sexism can be dismissed outright, if people find it safer to talk shit to a woman compared to a man.
What I wish the story had more of was how scientific criticism is done outside of Gelman's blog. Gelman is the main scientist/research I follow on the topic. Apparently some see his snarky critique as toxic, and Gelman says he is intending to reevaluate the way he writes about things so that his point doesn't get lost in cheap jokes. How do other blogs do it? I'm somewhat sympathetic to Gelman because I perceive that there is significant institutional resistance (and few incentives) to re-examine and correct a peer review study, and being harsh/edgy/rude and beating the drum over time might be the only way to get scientists to care? But that doesn't mean his critique can't be refined for the benefit of his mission.
Talking about the stakes is in direct conflict with the scientific ideal of focusing on the science without talking about its effects on the scientists involved.
For example, from Gelman's response: "I just wanted to emphasize that there’s no reason her career, or even her famous Ted talk, has to rely on a particular intriguing idea (on there being a large and predictable effect of a certain pose) that happened not to work out." Well, except that maybe she wouldn't have gotten a Ted talk without it?
He's saying, hey, this paper shouldn't have been as important as it was. And according to the scientific ideal, we shouldn't get too attached to any particular results. But in practice, putting bad results aside sometimes requires a significant change of identity. Forgetting about that, even for a moment, is always going to come across as tin-eared.
The scientific ideal is more compatible with low stakes than high stakes. But talking about the game as if it's low-stakes (or ignoring the stakes altogether) when actually it's high-stakes is always going to make discussions focused on the science alone come across as unsympathetic.
Science is about finding approximations of truth that are least influenced by the local context.
The whims of a particular scientist's ego (especially if you are that scientist!) make for a rather nasty contextual influence.
Of course, this is a difficult ask, but constantly striving towards this ideal is absolutely essential.
But does this make sense? Can a scientist survive without doing this? Can a scientist get very far without results right now?
Is this a problem of the scientist, or perhaps the social/economic expectation?
New results are significantly more difficult and expensive to achieve in modern science, as the easier stuff has already been conquered in centuries and decades past.
The harder problems that science now works on means that distinguishing between actual results and incorrect conclusions is also much more difficult.
The growth of science as a percentage of the population and economy has also finally slowed down a few decades ago, after centuries of rocket-like growth. It is now a maturing enterprise, instead of a high-growth startup.
This means increased competition for resources and mindshare from those wanting to do science. Not surprisingly, all proxy measures and heuristics for evaluating scientific success are thus more likely to be abused.
I don't think we have fully adapted to these fundamental changes.
Disclaimer: I'm not a historian or researcher, but I've heard aspects of the above from a variety of sources, and it makes a lot of sense to me.
Of course, that's much easier for people in STEM than it is for social scientists.
But you also can't prevent people from learning about how the world works: the rewards and penalties that will affect their careers.
Even if you could, teaching students a myth about what a career in science is like would basically be malpractice.
A recent BBC event aimed at solving problems faced by women in tech flew Amy Cuddy in to give design and professional advice to the participants. The BBC story repeatedly celebrated Cuddy for giving the "second most watched TED talk ever". It also talked up the "Wonder Woman power pose" as a way to change your life.
Notably not present at the event were Cuddy's co-authors Carney and Yap, who have been vastly more receptive to the replication question, and less quick to promote uncertain results in non-academic channels.
Scientific falsehoods are almost always more impressive than the truth - they aren't constrained to realistic effects. At a certain point it starts to look like we're actively rewarding people for doing bad research and then rewarding them further for refusing to correct course.
This makes lesser people working in dirtier subjects very uncomfortable: when he hits them they have nothing of substance to hit back with except generic insults.
http://andrewgelman.com/2016/05/05/npr-bites/
http://andrewgelman.com/2016/11/03/its-not-about-reality-its...
I just read his blog for the first time and thought he’s a huge asshole. It seems to me the primary effect of science-shaming on the blog is to rally the pitchforks, something he can’t possibly be ignorant of. But he apparently keeps on doing it. No doubt it’s good for the hits, but doesn’t contribute to a dynamic that leads to good science. Oh wells.
A little needling on Gelman's blog is not too harsh, all things considered.
Sexism comes out in target choice, and the strength of which ones chases their prey, and the vehemence of attacks.
I do strongly believe that Cuddy was overly punished for being a woman, and being prominent in the field. Everyone loves when the "high and might" come "crashing down" -> the underdog stories and the desire for retribution are sought after more powerfully than (a) justified and (b) end worthiness of the goal.
I don't think anyone can honestly say that this has improved the field and made science better. It just means now that there is money and recognition to be made being attack dogs. That's hardly better.
As for Gelman, he wasn't portrayed in a great light, but I think it is fair. He's willing to encourage incivility and outright sexism and hostility as long as he doesnt have to talk to someone he disagrees with. That's the baseline definition of a coward there. And yes, I would discuss that with him in person and call him such a thing to his face.
Theres a pattern of successful women receiving extreme levels of hate (Kathy Sierra et al). I'm not aware of this happening to guys.
Can you be more specific? Or do you mean this in a general sense? Because isn't this obvious ignorance if so?
What I mean is that I haven't heard of guys receiving death, rape etc threats accompanied by doxxing (ie 'this is his address, let's go get him') at the levels I've heard about with women[0].
But what I'm doing here is drawing parallels between large groups of people self-organizing against a prominent, successful person.
It's interesting that (except for Adam Orth) the men involved, who were factually the victims of massive harassment, were portrayed as the villains by most of the media for minor infractions of progressive political values.
Edit: Just found another, an electoral college member who received death threats for voting for Trump (as he was required to).
http://thehill.com/blogs/in-the-know/in-the-know/308226-elec...
That's similar to the situation with sexual harassment, sexual assault, and domestic violence.
It's not that men aren't often the victims of these things (studies report that they are, at nearly the same rate as women[1, 2]). It's that men rarely report these things, and when they do, receive little sympathy, coverage, or justice.
1: http://www.slate.com/articles/double_x/doublex/2014/04/male_...
2: http://domesticviolencestatistics.org/men-the-overlooked-vic...
Given the level of heat these discussions arouse, I understand the comments re modding in the item earlier today: https://news.ycombinator.com/item?id=15507821
Based on private conversations with mutual friends, the reason Cuddy received so much heat is that she continues to profit off the mistaken findings and refuses to disavow them, despite overwhelming evidence.
She embodies everything wrong with the popular discussion of science.
The only analogue I can think of is Brian Wansink, whose scandal is much newer; who did not go nearly so far to profit from his own results; and who has received plenty of heat so far.
"Although 11 new papers have recently been published that do not show the downstream effects of power posing on behaviors, Cuddy is still fighting for power posing."
> She mentioned, at a psychology conference where she was presenting her work, that a study had recently been conducted on power posing. “They found no hormonal effects,” she said before taking a breath. “That study is done very well, and I trust those results.” Although 11 new papers have recently been published that do not show the downstream effects of power posing on behaviors, Cuddy is still fighting for power posing. The research, she says, still shows its effect on feelings of power: At the conference, she presented a comprehensive meta-analysis, a version of which, she says, she will soon publish, with a strong P-curve supporting that, and she also presented a P-curve suggesting that power posing had a robust effect on self-evaluations, emotions and moods.
Gelman regularly eviscerates the research of Brian Wansink on his blog, in the exact same way that he did to Cuddy.
I agree that this is true, but at a certain point it seems like a justified reaction to unequal importance.
Other Gelman targets like the "himmicanes" paper generated a bit of media attention without any real consequence. Power poses, meanwhile, have become a major pop-science hit. They're still being widely read and promoted, even by tech conferences and news orgs like the BBC. And Cuddy in particular has chosen an aggressive non-scientific promotion campaign while her co-authors have made more honest attempts to take on the replication question - if Carvey was getting the same amount of flak I'd be more concerned that something unfair was happening.
If power poses don't work, it seems like a much worse indictment of academic social psych than most other failures to replicate - the results were widely touted and are still being raised to public acceptance even as they fall apart. The challenges to stereotype threat, priming, and the IAT are even scarier - those results have become well-established and even enshrined in law.
Throughout the replication crisis, my greatest frustration has been the inconsistent narrative coming from researchers. They're simultaneously demanding formal academic critiques instead of blogs, ignoring those critiques when they do exist, and then replying with the same sort of informal aggression they decried in the first place.
I think you're exactly right that Gelman's aggressive approach is motivated by the field's unwillingness to acknowledge anything milder. And seeing responses like Fiske, Cuddy, and Wansink have largely convinced me that there's a major double-standard at work between the level of seriousness applied when attacking critics and the level applied when presenting results as academic truth.
They're in the same boat. It is maddening how much stereotype threat gets brought up to defend certain ideas, or how it has worked its way into corporate diversity policy, and it appears to be built on a foundation of sand [1]. The worst part is, if you question it publicly, you stand a change of being treated like James Damore.
[1] - https://www.psychologytoday.com/blog/rabble-rouser/201512/is...
It such a shame for you to start viciously attacking Cuddy again for positions she clearly doesn't hold, when this article is directly about your misconceptions.
Matter of fact I'd guess its comments like yours which are fueling the underlying issue. The usual issues with communicating emotion on the internet.
There should not be schisms on science, yet there are - camps on facts. This is a problem with far worse implications than replicability.
I love (genuinely, non-ironically) that you reconsidered the merits of the article after reading Gelman's positive assessment of it. A possibly underappreciated benefit of Gelman's unabashed approach to criticism is that when he writes "I thought Dominus’s article was excellent and fair to all sides", it's reasonable to assume that he actually believes this rather than just pretending to "make nice" with scripted double-talk. His willingness to criticize things that he sees as wrong makes it possible (for me, and apparently you) to trust his stated opinion of things he sees as right.
And I don't think sexism can be dismissed outright, if people find it safer to talk shit to a woman compared to a man.
You prefaced it with an "if", but do you think it is common that "people find it safer to talk shit to a woman compared to a man"? I'd guess that at this point most male academics would be significantly more reluctant to criticise in print the work of a female colleague than that of a fellow male. I see Gelman as trying to adhere (possibly recklessly) to a gender-blind approach to criticism, and find this refreshing.
I don't know whether they find it safer, but they certainly do it more, and are conditioned to do so from childhood.
"Although 11 new papers have recently been published that do not show the downstream effects of power posing on behaviors, Cuddy is still fighting for power posing."
This is an astonishing number of replications to be published on one topic in psychology. 2-3x the next highest number I heave heard of.
The placement of that sentence, plus the flattering photos, leads me to suspect a publicist was able to place a thumb on the scale.
she is "still fighting" for power posing but only as a useful technique which sometimes works for some people in some cases, as even the replicated studies seem to show improvements in subjective measures of "confidence", whatever that means.
Look at the reach of people like Malcolm Gladwell and stuff like his books and podcast, and NPR's Hidden Brain, which subsist on recounting these "surprising studies" and speculating about their profound meaning for society, to gigantic audiences. And they're wrong! Made up ideological BS with the gravitas of 'serious science' behind them.
Power posing, stereotype threat, the bystander effect, implicit bias. So many popular hypotheses have snuck into the popular culture and they are wrong. They're based on fudging numbers and statistics and throwing out results that don't fit the hypothesis. They deserve to be combated in the public sphere. They don't deserve to be shielded to protect the careers of the hucksters.
The scandal is that Cuddy's work met the standard of the bulk of published social science until lately. She studied the same phlogiston with the same tools as her colleagues. But when the revolution that comes for them, gently, is an epistemic advance, their pain is our gain. Join the revolution, comrad Cuddy!
> But the email that Simmons and Simonsohn had sent was, in fact, ambiguous: They had explicitly told her to drop the P-curve and yet left the impression that the paper was otherwise sound. At my request, Simmons looked back at his original email. I watched as he read it over. “Oh, yeah,” he said quietly. He had a pained look on his face. “We did say to drop the graph, didn’t we?” He read it over again, then sat back. “I didn’t remember that. This may be a big misunderstanding about — that email is too polite.” Cuddy and Carney had taken their advice literally.
(The NYT begins the metaphor with "revolution", parent extends it with "violent revolution", and I switched out one form of violence with another).
In the cutthroat world of academic soft sciences you're not getting tenure unless you make a breakthru and you're allowed zero mistakes. Its up or out. She made her one mistake, and she's out. And there's 10, 50, 5000 people equally qualified without opportunity to cheer on her demise. One of hundreds or thousands of equally qualified researchers now gets a chance at tenure, good for them bad for her.
On the other hand in the commercial software development world, we usually don't blacklist software devs from the field after their first bug because 1) we're low on supplies of software devs plus or minus propaganda, at least we don't have the excess of her field 2) the experience of having a software bug makes a better dev.
You can't expect non-pathological behavior from a pathological situation. Until the supply/demand hiring issues in academia are fixed, stories about academia are going to read like an insane elementary school playground version of Lord of the Flies. You can't produce 10 to 100 times as much product (new grads) as the market can adsorb without behavior for the few good positions being viciously cutthroat and extremely dramatic.
It's just curious to me that she and Dr Fiske wouldn't catch this crucial difference immediately. Effectively it turns power poses into little more than 'the power of positive thinking'.
That said, personal attacks and especially those focusing on her brain function really need to stop. Psychologists treating each other like shit is not just ironic, it's a disgrace. If they can't even behave properly with each other, how can we expect them to treat patients properly?
Meaning, say the power pose effect doesn't exist. But early on, it's exciting and new, and there's a placebo effect (which is fine!), but people also think it's real.
But then studies get replicated, and there's well-known doubt on whether the effect is real in a non-placebo manner. There might still be a placebo effect, but does it shrink?
And then there's a lot of controversy and the study is dragged through the mud, and power poses are mocked. Might the placebo effect be lessened even further (or obliterated) from this?
Could this be true even if it was originally valid and not just placebo?
For the time being, the answer is probably "if it works for you, then keep doing it." If power posing before an important meeting is how you psyche yourself up to be unafraid and effective in the meeting, then why not keep doing it, even if it doesn't seem to work for everyone?
Knowing the replication problems, I won't suggest power posing as a general thing. Maybe I will tell someone to try a bunch of things including power posing, and tell them to keep the things that they think are working.
In their "open peer review", the psychologists seemed to have all the class of a bunch of redditors. An uncomfortable process is made worse! Instead of motivating people to publish lame results for tenure, this new situation seems to motivate people to act like Internet trolls or tabloid journalists.
A more critical research community should help advance the field, but these researchers set a bad example.
So I'll give my opinion as a datascientist: null hypothesis significance testing is broken, the whole thing needs to get chucked. It's not fixable, and it's not only p-hacking that's the problem. Go read Frank Harrell, I like his writings on the topic more than Gelman's.
This is the same Susan Fiske who called people who pointed out statistical errors in psychology papers "methodological terrorists". http://andrewgelman.com/2016/09/21/what-has-happened-down-he... The apple does not fall far from the tree.
She is criticizing "unfiltered trash-talk [on social media]... [and] online vigilantes [who] are attacking individuals, their research programs, and their careers".
It's the same tension the article is referring to, between civility and public discourse. On one hand, you'd really like an open dialog about the quality of evidence presented. On the other hand, there's a real price to be paid when people are too worried about public humiliation to risk doing science on certain questions or working with certain people.
As the quote from Jay Van Bavel put it, "It's become like politics — we’ve created two camps of people who shouldn’t be in two camps in the first place"
This point resonated with me even though I'm on your side of those politics. What effect would it have on the OSS community if any even slightly bad code you write could end up in a hit piece published by The Atlantic, and if fact you witnessed people on Facebook saying your colleague, who maybe did write some kinda bad code that one time but is on the whole competent, is a charlatan who doesn't deserve the title of "programmer"?
I don't see as wide of a gap in effect on career between open source contributions for engineers and published papers for scientists as you seem to be implying
Her work was interesting no doubt, and in our scientific age we have always celebrated the novelty and originality of the scientific geniuses that have advanced our understanding.
But the real value of science has always been novelty plus certainty. The scientists who we esteem should be known as much for their industrious management of the empirical process (recruiting replicators, statistical auditors, etc.) as they are for they are for their creative insight.
geofft's comment hits the nail on the head when it comes to what effect this did vs. should've had on her tenure case, I think. A study that fails to replicate is no indication of malice or even incompetence. Some say she should've run self-replications, but how many years self-replication studies are you willing to run on $20k/yr (in fact, how many post-college years have you made 20k/yr for a full-time job)?
At some point you get the quality of science you can pay for, and throwing the grad student under the bus by denying them tenure a decade later is as shitty and spineless as Equifax throwing an IT guy under the bus for what was pretty clearly an organizational failure. IMO.
Re: TED talks, TED is not a scientific venue, and most of the things I hear in TED talks are a hell of a lot less scientific than even Cuddy's most flawed studies. Also, I don't think those sorts of things should weigh much in hiring and tenure cases either way (aside from e.g. a usually marginal "service/outreach" category).
If Harvard hired Cuddy because she gave a popular TED talk, then that was an extremely poor hiring decision by Harvard. But that's a separate issue from the one we're discussing.
I know, and I think we are mostly in agreement. I generally don't like seeing individuals dragged through the mud. It's also somewhat unfair to judge her actions pre-replication crisis by our standards post-replication crisis.
I also agree that a single study failing to replicate is not an instance of malice or incompetence, in fact I think all scientists should expect it to happen. But that also means that no career should be built on the back of a single study. It seems that she is still sticking to her guns even after 11 failed replications. That does cause me some concern.
> but how many years self-replication studies are you willing to run on $20k/yr? At some point you get the quality of science you can pay for...
Then maybe one should more hesitant about promoting those results?
> If Harvard hired Cuddy because she gave a popular TED talk, then that was an extremely poor hiring decision by Harvard. But that's a separate issue from the one we're discussing.
I think it makes me more sympathetic to her case. If Harvard had not hired her, or TED not promoted her, she would not have been subject to nearly the same scrutiny or ire as she was. Ironically, she might have had a better shot at a good long-term career. But who, at that point, would have turned down either of these prizes? I know I wouldn't have.
We definitely agree on this. I think all the TED science stuff is silliness. Cuddy's mistake was both being a serious scientist and being a popular TED speaker. Popular TED speakers make BS science claims all the time, but they aren't scientists so it doesn't matter.
But I wouldn't advocate for firing someone because they disagree and think TED is a good place to popularize science (even if they happen to popularize science that later turns out to be bad science).
There is a concept in management called the Peter principle, which is that "managers rise to the level of their incompetence". I've always worried about it in myself - I work hard to advance my career, but is it possible I'm sabotaging my own long term success? What if I get the high-profile new responsibilities I'm striving towards... and then fail spectacularly?
> It's given me the impression that if I don't create something that is up to a teams standard from commit 0
Linus has said that he reserves his abuse for senior people who should know better. It's his antidote to a complacent overclass that mistreats the kernel development process.
Let me quote Gelman regarding the subject of her criticism:
> The other thing that’s sad here is how Fiske seems to have felt the need to compromise her own principles here. She deplores “unfiltered trash talk,” “unmoderated attacks” and “adversarial viciousness” and insists on the importance of “editorial oversight and peer review.” According to Fiske, criticisms should be “most often in private with a chance to improve (peer review), or at least in moderated exchanges (curated comments and rebuttals).” And she writes of “scientific standards, ethical norms, and mutual respect.”
> But Fiske expresses these views in an unvetted attack in an unmoderated forum with no peer review or opportunity for comments or rebuttals, meanwhile referring to her unnamed adversaries as “methological terrorists.” Sounds like unfiltered trash talk to me. But, then again, I haven’t seen Fiske on the basketball court so I really have no idea what she sounds like when she’s really trash talkin’.
> I bring this up not in the spirit of gotcha, but rather to emphasize what a difficult position Fiske is in. She’s seeing her professional world collapsing—not at a personal level, I assume she’ll keep her title as the Eugene Higgins Professor of Psychology and Professor of Public Affairs at Princeton University for as long as she wants—but her work and the work of her friends and colleagues is being questioned in a way that no one could’ve imagined ten years ago. It’s scary, and it’s gotta be a lot easier for her to blame some unnamed “terrorists” than to confront the gaps in her own understanding of research methods.
> To put it another way, Fiske and her friends and students followed a certain path which has given them fame, fortune, and acclaim. Question the path, and you question the legitimacy of all that came from it. And that can’t be pleasant.
I'm less concerned with Fiske vs. Gelman and much more concerned with Jay Van Bavel's observation that either way, science loses.
Confining scientific critique to scientific venues seems like a reasonable bulwark against this sort of thing. If it had been Reviewer #3 in a program committee or editorial board, or a well-received letter to the editor, instead of a blog post picked up by thousands of internet warriors with no stats training, perhaps the result could've been refined or thrown out without expelling someone from research. If the feud hadn't been picked up by pop science press, then maybe a bad piece of science could've been questioned without poisoning the well for further inquiry.
Fiske controls the venue and is (mis-)using the "scientific venue" to denounce her critics. She is the one refusing to publish the very critiques you describe. The "scientific venue" has been corrupted. The well has not been "poisoned" for further inquiry, rather the necessity of replication and sensitivity to statistical errors has increased. Science was being done sloppily, and Fiske was both a beneficiary of that and in a role where she could have prevented it.
That's a pretty strong accusation. Do you have any evidence?
Cuddy herself, in the linked article, even asks: "Why not come to a conference or hold a seminar?"
I don't think you're wrong about the lack of quality reviewing, but I also think that blogs and Facebook and articles in pop sci news venues are pretty shitty places to address the quality of editorial review.
And your post demonstrates why: honest mistakes -- or even complete non-mistakes stemming from gaps in established best practices! -- turn into out-right accusations of full on corruption. The comments on NYT demonstrate the same problem from the other side: lots of gender politics when Gelman wasn't motivated by gender, implicitly or otherwise, but rather by the mass appeal of Cuddy's work.
Needless to say, I also think you're massively over-stating the maliciousness of people like Cuddy and Fiske.
> Science was being done sloppily, and Fiske was both a beneficiary of that and in a role where she could have prevented it.
Simmons seems to agree with me that you're really over-reaching here: "The original study wasn’t particularly egregious," he said. "It was published in 2010 before anyone was thinking about this."
There's a huge difference between doing science the way you were trained (but sub-optimally), and outright "corruption".
There's also a HUGE difference between noticing a problem and constructively pushing for more rigor within a sub-community, and going after particular people within that community so harshly you destroy their careers.
The former is great for science. The latter might possibly by great for improving rigor (I'm pessimistic and think it all comes down to incentives, which Gelman et al don't really do much to change). But equivocating bad methodological/statistical training/practice with unethical and even criminal ("corrupt") behavior? That sort of dishonest fire-and-brimstone talk is just as damaging as a lack of rigor.
1. failing to have read Meehl's critique doesn't amount to "corruption",
2. that calling people "corrupt" for having failed to read Meehl is not likely to be the most effective or the most civil path toward changing things, and
3. To the extent that discussion of scientific methodology on public forums causes or contributes to (2), that's a bad thing and perhaps suggests a fundamental cost of doing science in this manner.
I'm having trouble finding a charitable way to read the claim that someone is misusing a scientific venue.
And much more to the point, I doubt even more that the subject of those sentences (or the e.g. tenure/promotion committee of that subject) could be reasonably expected to apply such an abundance of charity.
In a sense, both pieces are self-published, and she is using the mantle of Science to falsely claim that her self-publishing is superior because of the authority of her position as editor. This is corruption. This is misusing the scientific venue.
Please link to the place where she actually makes this claim.
> in her journal
Not hers, and not a journal.
> while his self-published piece is "methodological terrorism."
She actually outlines the practices that ought to be singled out as unprofessional and unacceptable in the second to last paragraph, and none of those are mere self-publication.
Have you considered the possibility that both sides are wrong? That this is crap science, and that the conduct of certain groups of people rallying against crap science is overzealous to the point of cruelty?
Finally, it's also worth noting you've COMPLETELY proven Fiske's point about social media: sure, there are some valid critiques. But there are also people being absolutely vicious to people actually trying to science because of very trivial and obvious misunderstandings (e.g., because they think that APS Observer is a journal). I think to some extent that's a reasonable price for free discourse when things stay online and especially when they're anonymous, but you're calling a real person with a real job on the line corrupt because you made an incorrect assumption (that would've taken you 10 seconds of googling to correct).
"Cuddy herself, in the linked article, even asks: "Why not come to a conference or hold a seminar?""
I am not criticizing Cuddy. Fiske is editor of PNAS, former president of APS, and is putting her criticisms in APS Observer. The difference between a journal and an official magazine of a scientific organization is irrelevant in this concept, in that the supposed superiority of an edited journal is what is in question here when the editor is not good, and is not considering the context of the discussion.
I am pointing out the utter hypocrisy in her argument that you shouldn't point out statistical errors on blogs, while she is pointing this out in a publication she controlled. She wants people to be able to protect their undeserved status, it is clear.
The entire idea that science needs to be locked up in limited access journals (or magazines) is not a good one. The real issue is that the quality of PNAS has been pretty low. Fiske's response to that is to try to discourage open criticism. Open data is one of the reasons that people were able to find these errors. Resistance to publishing failed replications is not proper behavior. We have now seen PNAS publish a few letters describing errors, and hopefully the standards will improve. However, it hardly seems like a person whose response is to lean toward hiding the discourse is the right person to do it?
Fiske is the journal editor. Fiske is allowing articles with weak stats to get published. Fiske is publishing her defense in that journal. Fiske is not publishing critical articles in that journal. That's corruption based on her position of authority.
Cuddy just had sloppy stats and her results failed to replicate. Notice how you deceptively switch to Cuddy from Fiske in your response, because you have no point about Fiske.
APS Observer is a magazine, not a journal.
Furthermore, Fiske was not the president of APS or an editor of the Magazine when that article was published. And even if she were, what's the problem? I see letters from the editor in most magazines I read on a very regular basis.
Finally, the claim that Fiske was construing her opinion on conduct as a scientific fact is just... well.
I'll refrain from calling you corrupt or deceptive because it was probably just a misunderstanding :-)
> Notice how you deceptively switch to Cuddy from Fiske
I actually have no idea what you're trying to say or what this refers to. Can you be more explicit?
I don't think turning a call for greater scientific rigor into a political fight is desirable, but neither do I think that confining this struggle to scientific venues is desirable (or practical, for that matter).
EDIT: wording fix
When you take your BS "studies" and sell it to the audience at TED and then the wider internet, go on corporate speaking tours, sign book deals to promote your "science", and try to affect public policy, you are open to criticism from anyone who sees through your shoddy methodology and wants to correct the record.
This is a story in which a few people look to be "right", but no one looks particularly flattering, which is also why its a good read. Your comment adds very little.
The co-author of this study, Dana Carney, distanced herself from this study [1]. She doesn't believe all the claims the study made were true.
[1] http://www.npr.org/2016/10/01/496093672/power-poses-co-autho...
It's OK to be oppositional or confrontational to people based on what they say and do, but self-styled reformists sometimes become personally vindictive in pursuit of their own interests. Picking a fight with a public figure is an easy way to gain publicity, and if your own reputation rests on winning that fight then it can be tempting to carry on into harassment. Gamergate was an extreme example of this.