What has happened down here is the winds have changed

If I wrote code that messed up, I'd patch it. If someone else points out a bug? Even better, now I don't have to find it myself. But I would definitely not feel attacked or nor would I blame the QA engineer. It's my mistake, and so I must own it. These researchers need to do the same.

I believe full publication of code and data should be required for peer-reviewed research. If I can't look at your raw data or your algorithm, it is simply not possible to determine whether the study is correct. The social sciences must adapt to more statistically rigorous methods.

After all, the only two choices are adaptation or extinction.

Apparently (I gleaned from the comments) Susan was very vocally opposed to that project and said it was an attack.

She's painting a bleak picture of widespread social attacks and commentary causing the problems in academics that I don't think matches reality, and she's also suggesting the public shouldn't comment on publicly funded research.

I don't know of a single academic that has left the field due to negative commentary coming from outside the peer review system, and I know a lot of people that have left academics and a lot of people still in academics.

Some ambitious academic peers are vicious, and people are leaving due to infighting, trouble getting funding, academic stealing, difficulty getting tenure, and general departmental and university politics. The single biggest threat to education is our funding model and the exploding costs stemming from exploding amounts of administration.

The paper review process not only isn't one of the major problems, I personally believe it's actually working better now than it ever has, and that the quality and standards are higher than they've ever been.

The comments on Statcheck were wildly in favor of having automated bots fact-checking publications. I contend that it's best use is before publication, and not after, but otherwise I agree - it's awesome.

Both editorials are written by people in positions of academic power (Fiske is a PNAS editor - PNAS editors have a unique and often criticized ability to unilaterally review and approve publication of articles without a peer review panel; NEJM editors wield great power in medical sciences).

Both are originally directed toward a narrower audience of their journal, but taken out of the confines of their academic cloister, start to sound ridiculous in a world where the public starts to point out where their funding is coming from, and poke holes in their reasoning.

Secondly, addressing the article's main point, I'm being told that social media comments about their studies are so traumatic and detrimental that they leave their careers. That's a problem with the researcher's ability to handle digital hate mail. Doctors, lawyers, dentists, and veterinarians all get massive amounts of digital hate mail.

Finally, after the failure to replicate studies, "self-appointed data police" and "methodological terrorism"[0] sound like exactly what the field needs.

[0] I have to say, only an incredibly insular community could come up with such an insensitive name. Checking your methods is not terrorism. Someone doing preliminary reviews on research data is not terrorism. To compare data critics to terrorists both insults the victims of actual terrorism, and dilutes the definition of the term so far as to be almost meaningless. They may as well call them "Methodological Nazis."

The work of psychology researchers influences our LAWS. They influence how I can live my life. It's a very personal thing. I will criticise them however I want and as much as I want until this branch of 'science' has cleaned up its conduct.

If your work cannot withstand criticism, it's worthless. Get out of science, Susan T. Fiske, you give it a bad name. Or rather, what you were doing was probably never science in the first place. Your attitude fills me with disgust.

This is not a Russel's Paradox situation that can be patched up. The crisis in social 'sciences' is of such a magnitude that the only reasonable course of action is "start over" in many cases.

"In the rest of society, however, we often both try to hire people who seem to show off the highest related abilities, and we let those most prestigious people have a lot of discretion in how the job is structured. For example, we let the most prestigious doctors tell us how medicine should be run, the most prestigious lawyers tells us how law should be run, the most prestigious finance professionals tell us how the financial system should work, and the most prestigious academics tell us how to run schools and research.

This can go very wrong! Imagine that we wanted research progress, and that we let the most prestigious researchers pick research topics and methods. To show off their abilities, they may pick topics and methods that most reduce the noise in estimating abilities. For example, they may pick mathematical methods, and topics that are well suited to such methods. And many of them may crowd around the same few topics, like runners at a race. These choices would succeed in helping the most able researchers to show that they are in fact the most able. But the actual research that results might not be very useful at producing research progress.

Of course if we don’t really care about research progress, or students learning, or medical effectiveness, etc., if what we mainly care about is just affiliating with the most impressive folks, well then all this isn’t much of a problem. But if we do care about these things, then unthinkingly presuming that the most prestigious people are the best to tell us how to do things, that can go very very wrong."

The winds may have changed, but landscape erodes very slowly…

[0] http://www.overcomingbias.com/2016/06/beware-prestige-based-...

We live in an era where authorities of all kinds are under greater scrutiny than ever before, because the Internet's made it so much easier for everyone to learn about and discuss what they're doing. And to believe that we know better than them. The mass media gatekeepers who once shaped public opinion are still present but have a fraction of the influence they had in the 20th century.

Nassim Taleb discussed this recently from a more political angle: https://medium.com/@nntaleb/the-intellectual-yet-idiot-13211...

But it is really the same thing. From the post-Snowden privacy movement to Brexit to the Trump campaign to "the destructo-critics" referred to in this article, all of these things are a product of people becoming more informed about what authorities are really doing, getting pissed off about it, and taking action.

You can make your own value judgments about whether all of these things are good or bad (we surely know what the predominant opinions will be among HN's audience of left wing tech people: pro-Snowden, anti-Brexit, anti-Trump, pro destructo-critics). But I think the overall trend is good. And intensely destabilizing to society, in no small part because even as we approach this information equality, wealth inequality is soaring.

I think one important thing to understand about this trend is that _numbers win_ -- you can be right but if a large enough part of the population disagrees with you, you're going to be on the defensive anyway.

Another thing is that while people are rapidly becoming better informed, they're becoming more confident in their opinions even faster. So again we are not necessarily always going to see better decisions being made -- but they are going to be made with more public input.

I know that might sound a little crazy when we seemingly hear about a new secret conspiracy or abuse of power every day, but that's the point: _this stuff was always going on and we just didn't know._

As Feynman said: "We've learned from experience that the truth will come out. Other experimenters will repeat your experiment and find out whether you were wrong or right. Nature's phenomena will agree or they'll disagree with your theory. And, although you may gain some temporary fame and excitement, you will not gain a good reputation as a scientist if you haven't tried to be very careful in this kind of work. And it's this type of integrity, this kind of care not to fool yourself, that is missing to a large extent in much of the research in cargo cult science."

And I agree with the author that it shouldn't.

A good point is made about the arguments for releasing study findings in blogs, where discussions can be had in comments and perhaps the content of the blog amended as needed.

I definitely think that the old way of publishing studies needs to change. They need to allow discussion. Those that publish the studies and those that read them need to critique and possibly conclusions be amended. And to get this to happen at an intellectual level, students and others in academia and research should be funded more by taxes and possibly be required to do critiques and duplicate other studies to some extent.

For too long we've assumed the bodies of accumulated knowledge in the sciences to be fact without dispute; that's not science. Science is all about finding models that work and gathering data and trying to draw conclusions from it. Science was never meant to be about facts; it is all about understanding. Though some understanding may be innate, much of human understanding comes from experience. While we can build upon previous "knowledge", time over time we've seen what we held to be science fact proven false, e.g. earth is flat -> it's round but the sun goes around it -> it goes around the sun, and flies spontaneous generate from feces -> flies lay eggs in feces. If we'd never allow criticism, we'd never have progressed.

Science should be about keeping an open mind and accepting there is much we don't know, even if you believe and have faith in something. Einstein believed in God, and Hawking is an atheist. Both are scientists, and both have had theories proven and disproven. We are human, and we must continually strive for understanding knowing some ideas may be right, some may be wrong, and that ideas will change.

Realizing this is a good thing. It's going to set back careers and reduce funding, though.

Next, economics?

"Science is prediction, not explanation." - Fred Hoyle

https://youtu.be/MGs2iLoDUYE

If you only know about him as the guy who does Disney/Pixar music you're really missing out.

> In retrospect, Bem’s paper had huge, obvious multiple comparisons problems—the editor and his four reviewers just didn’t know what to look for—but back in 2011 we weren’t so good at noticing this sort of thing.

I was a postdoc in a Psychology department when this was going on, and "obvious multiple comparisons problems" isn't a good characterization. Any competent psychology researcher in 2011 (a) understood multiple comparisons and looked for them as a matter of course (b) knew there was something wrong with Bem's paper (see the editorial disclaimer).

Here is the main takedown of it: https://dl.dropboxusercontent.com/u/1018886/Bem6.pdf

That is some pretty advanced statistics, not just "correct for multiple comparisons".

What was ongoing then, and continues now, is that psychology and social science in general is coming around to the realization that the tools of the past 50 years are flawed, and to correct them, they need to become better statisticians. But it isn't a matter of "take stats 101 noobs", these are people who have been doing statistical analysis routinely for years. I think there is anxiety that to really do things right you need to _primarily be_ a statistician.

So there is some defensiveness in social sciences about this, certainly not helped by the fact that every jackass on the internet whose taken an undergrad math class thinks they know better.

In the end I quit my psych research job to be a software engineer since all the stats hurt my head and I needed something less quantitative to do.

And don't get illuded about your own field: Pretty much all of this is also true for computer science when it comes to quantitative research. How often have I seen studies like "we have tested this with 10 users which we divided in two groups".

https://en.m.wikipedia.org/wiki/Susan_Fiske A recent quantitative analysis identifies her [Susan Fiske] as the 22nd most eminent researcher in the modern era of psychology (12th among living researchers, 2nd among women).

^ Diener, E., Oishi, S., & Park, J. (in press). An incomplete list of eminent psychologists in the modern era. Archives of Scientific Psychology

We have so many "industries" now whose function is to exploit the taxpayer. We wouldn't have all this bad science or bad risk taking if there weren't a huge patsy willing to pay for it all without regard to actual results.

This doesn't at all match Fiske's article. As people on HN know very well, there's a difference between, on one hand, serious, constructive criticism; and on the other, the trolling, abuse, and just endless barrage of nonsense that buries any signal in noise, all of which is common online. It's clear Fiske is talking about the latter; the following is very recognizable:

... uncurated, unfiltered trash-talk, In the most extreme examples, online vigilantes are attacking individuals. Self-appointed data police are volunteering critiques of such personal ferocity and relentless frequency that they resemble a denial-of-service attack ...

It's not discussion, it's abuse, and we know well that the latter have tried to hide behind the former as a justification. And we also are very familiar now with the consequences:

Only what's crashing are people. The unmoderated attacks create collateral damage to targets' careers and well-being, with no accountability for the bullies. Our colleagues at all career stages are leaving the field because of sheer adversarial viciousness.

----

Then our author fabricates and attributes ideas to Fiske that she never mentions or even alludes to:

> She’s implicitly following what I’ve sometimes called the research incumbency rule: that, once an article is published in some approved venue, it should be taken as truth.

The word 'implicitly' is not a license to fabricate the rest of the sentence. And he goes on to criticize the whole field, despite saying at the beginning,

> it’s pretty much all about internal goings-on within the field of psychology (careers, tenure, smear tactics, people trying to protect their labs, public-speaking sponsors, career-stage vulnerability), and I don’t know anything about this, as I’m an outsider to psychology and I’ve seen very little of this sort of thing ...

I don't know anything about this; enough said. Also, his characterization of her article makes me wonder if he read it; her article is about trolling and abuse, and the things he mentions are only second-order effects.

The above behaviors are all familiar, and also telling is another habit we'll recognize:

> I was inclined to just keep my response short and sweet, but then it seemed worth the trouble to give some context.

Maybe we found one of the trolls.

Remember that whole cultural appropriation thing last year? Or the safe space movements on campus? There are plenty of movements that abuse political correctness to bully anyone around. On the opposite side there are plenty of 4chan types who love to shamelessly bully anyone who dare come out with a feministic message.

I imagine academics in related fields are ripe for being targeted by Internet bullies and trolls.

Isn't it this kind of abuse that Fiske is referring to?

" In short, Fiske doesn’t like when people use social media to publish negative comments on published research. She’s implicitly following what I’ve sometimes called the research incumbency rule: that, once an article is published in some approved venue, it should be taken as truth. "

after citing an article of Susan Fiske. The sentences in quotes are blatant lies, obvious if the cited article is read. She says, in the cited article:

" In contrast, the self-appointed destructive critic's role now includes public shaming and blaming, often implying dishonesty on the part of the target and other innuendo based on unchecked assumptions. Targets often seem to be chosen for scientifically irrelevant reasons: their contrary opinions, professional prominence or career stage vulnerability. "

Shame on you Mr. Gelman.

Go spend time on any politics forum for example and it is downright scary. Conspiracies run rampant and are unchecked, experts are personally vilified and viciously attacked and facts come second to feelings and opinions. And all evidence to date suggests that this is not helping society but actually making it more polarised and less cohesive.

And exactly the same thing has happened in other fields e.g. climate change. Legitimate criticism is always useful and welcomed but it needs to be constructive.

IMO this is not specific to psychology: climate change, biology, computer science, physics, etc. research is exposed to the same phenomena. The hotter the topic, the bigger the flames get.

However, if publication is clear about methodology, data collection and conclusions those flames do not hurt -- the published work stands on its own and anyone can go back to it as a sanity check on whether accusations have merit. If I disagree on methodology I will not think worse about the author. However, if I suspect he manipulated the data I will; and real quick.

Being clear on what was done is, IMO, a practically foolproof way of protecting your reputation. Researchers unnecessarily making publications readable to only a tiny minority of field experts (sometimes hoping to minimize criticism -- "I just need this published; and 3 more for a postdoc") or being vague about how they collected data shoot themselves in the foot. And this, IMO, is a major source of self inflicted wounds that are most frequent in the fuzzier fields like psychology.