Comment by throwawayjava | Hacker News Reader

Comment by throwawayjava | Hacker News Reader

Parent Full thread

You mischaracterize. Fiske is not criticizing "people who pointed out statistical errors" and begins the entire essay by explicitly stating as such: "Our field has always encourages -- required, really -- peer critiques"!

She is criticizing "unfiltered trash-talk [on social media]... [and] online vigilantes [who] are attacking individuals, their research programs, and their careers".

It's the same tension the article is referring to, between civility and public discourse. On one hand, you'd really like an open dialog about the quality of evidence presented. On the other hand, there's a real price to be paid when people are too worried about public humiliation to risk doing science on certain questions or working with certain people.

As the quote from Jay Van Bavel put it, "It's become like politics — we’ve created two camps of people who shouldn’t be in two camps in the first place"

This point resonated with me even though I'm on your side of those politics. What effect would it have on the OSS community if any even slightly bad code you write could end up in a hit piece published by The Atlantic, and if fact you witnessed people on Facebook saying your colleague, who maybe did write some kinda bad code that one time but is on the whole competent, is a charlatan who doesn't deserve the title of "programmer"?

> You mischaracterize. Fiske is not criticizing "people who pointed out statistical errors" and begins the entire essay by explicitly stating as such: "Our field has always encourages -- required, really -- peer critiques"! > She is criticizing "unfiltered trash-talk [on social media]... [and] online vigilantes [who] are attacking individuals, their research programs, and their careers".

Let me quote Gelman regarding the subject of her criticism:

> The other thing that’s sad here is how Fiske seems to have felt the need to compromise her own principles here. She deplores “unfiltered trash talk,” “unmoderated attacks” and “adversarial viciousness” and insists on the importance of “editorial oversight and peer review.” According to Fiske, criticisms should be “most often in private with a chance to improve (peer review), or at least in moderated exchanges (curated comments and rebuttals).” And she writes of “scientific standards, ethical norms, and mutual respect.”

> But Fiske expresses these views in an unvetted attack in an unmoderated forum with no peer review or opportunity for comments or rebuttals, meanwhile referring to her unnamed adversaries as “methological terrorists.” Sounds like unfiltered trash talk to me. But, then again, I haven’t seen Fiske on the basketball court so I really have no idea what she sounds like when she’s really trash talkin’.

> I bring this up not in the spirit of gotcha, but rather to emphasize what a difficult position Fiske is in. She’s seeing her professional world collapsing—not at a personal level, I assume she’ll keep her title as the Eugene Higgins Professor of Psychology and Professor of Public Affairs at Princeton University for as long as she wants—but her work and the work of her friends and colleagues is being questioned in a way that no one could’ve imagined ten years ago. It’s scary, and it’s gotta be a lot easier for her to blame some unnamed “terrorists” than to confront the gaps in her own understanding of research methods.

> To put it another way, Fiske and her friends and students followed a certain path which has given them fame, fortune, and acclaim. Question the path, and you question the legitimacy of all that came from it. And that can’t be pleasant.

Gelman has forged his own path to "fame, fortune, and acclaim". It's not clear to me that this path has actually improved (this particular branch of) science. In fact, by the sound of it, that path may've been at least as destructive as he claims Fiske et al's was. Which isn't an indict of Gelman, as I hardly intuit anything but positive intentions on his part.

I'm less concerned with Fiske vs. Gelman and much more concerned with Jay Van Bavel's observation that either way, science loses.

Confining scientific critique to scientific venues seems like a reasonable bulwark against this sort of thing. If it had been Reviewer #3 in a program committee or editorial board, or a well-received letter to the editor, instead of a blog post picked up by thousands of internet warriors with no stats training, perhaps the result could've been refined or thrown out without expelling someone from research. If the feud hadn't been picked up by pop science press, then maybe a bad piece of science could've been questioned without poisoning the well for further inquiry.

"Confining scientific critique to scientific venues"

Fiske controls the venue and is (mis-)using the "scientific venue" to denounce her critics. She is the one refusing to publish the very critiques you describe. The "scientific venue" has been corrupted. The well has not been "poisoned" for further inquiry, rather the necessity of replication and sensitivity to statistical errors has increased. Science was being done sloppily, and Fiske was both a beneficiary of that and in a role where she could have prevented it.

> The "scientific venue" has been corrupted

That's a pretty strong accusation. Do you have any evidence?

Cuddy herself, in the linked article, even asks: "Why not come to a conference or hold a seminar?"

I don't think you're wrong about the lack of quality reviewing, but I also think that blogs and Facebook and articles in pop sci news venues are pretty shitty places to address the quality of editorial review.

And your post demonstrates why: honest mistakes -- or even complete non-mistakes stemming from gaps in established best practices! -- turn into out-right accusations of full on corruption. The comments on NYT demonstrate the same problem from the other side: lots of gender politics when Gelman wasn't motivated by gender, implicitly or otherwise, but rather by the mass appeal of Cuddy's work.

Needless to say, I also think you're massively over-stating the maliciousness of people like Cuddy and Fiske.

> Science was being done sloppily, and Fiske was both a beneficiary of that and in a role where she could have prevented it.

Simmons seems to agree with me that you're really over-reaching here: "The original study wasn’t particularly egregious," he said. "It was published in 2010 before anyone was thinking about this."

There's a huge difference between doing science the way you were trained (but sub-optimally), and outright "corruption".

There's also a HUGE difference between noticing a problem and constructively pushing for more rigor within a sub-community, and going after particular people within that community so harshly you destroy their careers.

The former is great for science. The latter might possibly by great for improving rigor (I'm pessimistic and think it all comes down to incentives, which Gelman et al don't really do much to change). But equivocating bad methodological/statistical training/practice with unethical and even criminal ("corrupt") behavior? That sort of dishonest fire-and-brimstone talk is just as damaging as a lack of rigor.

Just so you know, the basic statistical problem with this type of research was pointed out back in the 60's (http://andrewgelman.com/2016/05/06/needed-an-intellectual-hi...), it's just that nobody listened.

My point is that:

1. failing to have read Meehl's critique doesn't amount to "corruption",

2. that calling people "corrupt" for having failed to read Meehl is not likely to be the most effective or the most civil path toward changing things, and

3. To the extent that discussion of scientific methodology on public forums causes or contributes to (2), that's a bad thing and perhaps suggests a fundamental cost of doing science in this manner.

I think you're using a different sense of the word "corrupt" from the one in the post you're replying to ("The "scientific venue" has been corrupted."). I know it's easy to lose nuance in internet discussions, but please read the posts above more carefully/charitably. Nothing Cuddy did was corrupt in any sense, only poor science.

I re-read the comment, and disagree. The preceding sentence is: Fiske controls the venue and is (mis-)using the "scientific venue" to denounce her critics

I'm having trouble finding a charitable way to read the claim that someone is misusing a scientific venue.

And much more to the point, I doubt even more that the subject of those sentences (or the e.g. tenure/promotion committee of that subject) could be reasonably expected to apply such an abundance of charity.

Think "corrupted file", not "corrupt official". The key difference is that there is a primary purpose for their existence (a file stores data, a scientific process produces good and reliable findings), and something can be corrupted (no longer suitable for its purpose) without malice involved. To call an official corrupted, you need to allege malice, to say that a file is corrupted, you only need to show that it's not valid for its purpose. It's not a good choice of a word anyway, you could make exactly the same valid point without any of the ambiguity.

edit: My defense of my reading isn't necessary; just see OP's sibling post. I interpreted their post correctly.

Ugh, yeah.

How's this- Fiske publishes a letter critical of Gelman, with no peer review of said letter, in her journal. She then claims her letter is "science" because it is in a scientific journal, while his self-published piece is "methodological terrorism."

In a sense, both pieces are self-published, and she is using the mantle of Science to falsely claim that her self-publishing is superior because of the authority of her position as editor. This is corruption. This is misusing the scientific venue.

> She then claims her letter is "science"

Please link to the place where she actually makes this claim.

> in her journal

Not hers, and not a journal.

> while his self-published piece is "methodological terrorism."

She actually outlines the practices that ought to be singled out as unprofessional and unacceptable in the second to last paragraph, and none of those are mere self-publication.

Have you considered the possibility that both sides are wrong? That this is crap science, and that the conduct of certain groups of people rallying against crap science is overzealous to the point of cruelty?

Finally, it's also worth noting you've COMPLETELY proven Fiske's point about social media: sure, there are some valid critiques. But there are also people being absolutely vicious to people actually trying to science because of very trivial and obvious misunderstandings (e.g., because they think that APS Observer is a journal). I think to some extent that's a reasonable price for free discourse when things stay online and especially when they're anonymous, but you're calling a real person with a real job on the line corrupt because you made an incorrect assumption (that would've taken you 10 seconds of googling to correct).

I talk about the problems with Fiske and you say:

"Cuddy herself, in the linked article, even asks: "Why not come to a conference or hold a seminar?""

I am not criticizing Cuddy. Fiske is editor of PNAS, former president of APS, and is putting her criticisms in APS Observer. The difference between a journal and an official magazine of a scientific organization is irrelevant in this concept, in that the supposed superiority of an edited journal is what is in question here when the editor is not good, and is not considering the context of the discussion.

I am pointing out the utter hypocrisy in her argument that you shouldn't point out statistical errors on blogs, while she is pointing this out in a publication she controlled. She wants people to be able to protect their undeserved status, it is clear.

The entire idea that science needs to be locked up in limited access journals (or magazines) is not a good one. The real issue is that the quality of PNAS has been pretty low. Fiske's response to that is to try to discourage open criticism. Open data is one of the reasons that people were able to find these errors. Resistance to publishing failed replications is not proper behavior. We have now seen PNAS publish a few letters describing errors, and hopefully the standards will improve. However, it hardly seems like a person whose response is to lean toward hiding the discourse is the right person to do it?

Here's a paper she edited on corruption. The conditions are there: http://www.pnas.org/content/112/34/10651

Here's an interview with the brave researcher who asked her about this at a conference. Her quote "There is no mechanism in place that holds researchers accountable for practices that lead to a corrupted literature but cannot clearly be classified as fraud" is probably a less aggressive version of what I am saying.

"That's a pretty strong accusation. Do you have any evidence?"

Fiske is the journal editor. Fiske is allowing articles with weak stats to get published. Fiske is publishing her defense in that journal. Fiske is not publishing critical articles in that journal. That's corruption based on her position of authority.

Cuddy just had sloppy stats and her results failed to replicate. Notice how you deceptively switch to Cuddy from Fiske in your response, because you have no point about Fiske.

> Fiske is publishing her defense in that journal

APS Observer is a magazine, not a journal.

Furthermore, Fiske was not the president of APS or an editor of the Magazine when that article was published. And even if she were, what's the problem? I see letters from the editor in most magazines I read on a very regular basis.

Finally, the claim that Fiske was construing her opinion on conduct as a scientific fact is just... well.

I'll refrain from calling you corrupt or deceptive because it was probably just a misunderstanding :-)

> Notice how you deceptively switch to Cuddy from Fiske

I actually have no idea what you're trying to say or what this refers to. Can you be more explicit?

> Confining scientific critique to scientific venues seems like a reasonable bulwark against this sort of thing.

I don't think turning a call for greater scientific rigor into a political fight is desirable, but neither do I think that confining this struggle to scientific venues is desirable (or practical, for that matter).

EDIT: wording fix

If Cuddy's phony research got into TED talks and books and a lecture circuit, why should non-phony criticism be locked up in journals?

I vacuously agree with you. See other threads for my opinion on TED talks.

"Confining scientific critique to scientific venues"

When you take your BS "studies" and sell it to the audience at TED and then the wider internet, go on corporate speaking tours, sign book deals to promote your "science", and try to affect public policy, you are open to criticism from anyone who sees through your shoddy methodology and wants to correct the record.

Is that not the case anyway? I admit I have no experience contributing but that's from hearing things like how the linux development team runs and people like Sarah Sharp quitting over the abuse. It's given me the impression that if I don't create something that is up to a teams standard from commit 0 then I will be ridiculed in a public forum

Abuse from your own insular community re: a particular project is one thing; public abuse published in some of the most popular print media on the planet resulting in lost jobs/lost job opporunities is quite another.

People use their open source contributions as part of their resume when seeking jobs, correct? That sort of abuse would reach outside the group if you were showing off your contributions as a necessary part of your career.

I don't see as wide of a gap in effect on career between open source contributions for engineers and published papers for scientists as you seem to be implying

I think we are violently agreeing on the original question about whether that's bad for the OSS community, though?

I certainly have a fair degree of sympathy for Cuddy's losses, but should she have been given the accolades (TED talk, tenure track at Harvard, etc.) in the first place?

Her work was interesting no doubt, and in our scientific age we have always celebrated the novelty and originality of the scientific geniuses that have advanced our understanding.

But the real value of science has always been novelty plus certainty. The scientists who we esteem should be known as much for their industrious management of the empirical process (recruiting replicators, statistical auditors, etc.) as they are for they are for their creative insight.

As I indicated up-thread, I'm mostly on your side of these politics.

geofft's comment hits the nail on the head when it comes to what effect this did vs. should've had on her tenure case, I think. A study that fails to replicate is no indication of malice or even incompetence. Some say she should've run self-replications, but how many years self-replication studies are you willing to run on $20k/yr (in fact, how many post-college years have you made 20k/yr for a full-time job)?

At some point you get the quality of science you can pay for, and throwing the grad student under the bus by denying them tenure a decade later is as shitty and spineless as Equifax throwing an IT guy under the bus for what was pretty clearly an organizational failure. IMO.

Re: TED talks, TED is not a scientific venue, and most of the things I hear in TED talks are a hell of a lot less scientific than even Cuddy's most flawed studies. Also, I don't think those sorts of things should weigh much in hiring and tenure cases either way (aside from e.g. a usually marginal "service/outreach" category).

If Harvard hired Cuddy because she gave a popular TED talk, then that was an extremely poor hiring decision by Harvard. But that's a separate issue from the one we're discussing.

> As I indicated up-thread, I'm mostly on your side of these politics.

I know, and I think we are mostly in agreement. I generally don't like seeing individuals dragged through the mud. It's also somewhat unfair to judge her actions pre-replication crisis by our standards post-replication crisis.

I also agree that a single study failing to replicate is not an instance of malice or incompetence, in fact I think all scientists should expect it to happen. But that also means that no career should be built on the back of a single study. It seems that she is still sticking to her guns even after 11 failed replications. That does cause me some concern.

> but how many years self-replication studies are you willing to run on $20k/yr? At some point you get the quality of science you can pay for...

Then maybe one should more hesitant about promoting those results?

> If Harvard hired Cuddy because she gave a popular TED talk, then that was an extremely poor hiring decision by Harvard. But that's a separate issue from the one we're discussing.

I think it makes me more sympathetic to her case. If Harvard had not hired her, or TED not promoted her, she would not have been subject to nearly the same scrutiny or ire as she was. Ironically, she might have had a better shot at a good long-term career. But who, at that point, would have turned down either of these prizes? I know I wouldn't have.

> Then maybe one should more hesitant about promoting those results?

We definitely agree on this. I think all the TED science stuff is silliness. Cuddy's mistake was both being a serious scientist and being a popular TED speaker. Popular TED speakers make BS science claims all the time, but they aren't scientists so it doesn't matter.

But I wouldn't advocate for firing someone because they disagree and think TED is a good place to popularize science (even if they happen to popularize science that later turns out to be bad science).

Cuddy wasn't fired, or even denied tenure, though it's probably not a coincidence that she's no longer (or not currently) on the tenure track at HBS.

> I think it makes me more sympathetic to her case. If Harvard had not hired her, or TED not promoted her, she would not have been subject to nearly the same scrutiny or ire as she was. Ironically, she might have had a better shot at a good long-term career. But who, at that point, would have turned down either of these prizes? I know I wouldn't have.

There is a concept in management called the Peter principle, which is that "managers rise to the level of their incompetence". I've always worried about it in myself - I work hard to advance my career, but is it possible I'm sabotaging my own long term success? What if I get the high-profile new responsibilities I'm striving towards... and then fail spectacularly?

[0]: https://en.wikipedia.org/wiki/Peter_principle

(note: Havard _Business_ School, not any of the Harvard schools of sciences)

good point. I missed that at first.

I find it amazing and heartening to see this kind of humanistic, self-reflective discourse on hn, where I generally expect to find macho posturing.

Before deciding that Linux (the most successful open source projecting history?) is poorly run. you have to weigh people quitting over the abuse against all the other contributors and maintainers building something that relies in part on that abuse.

> It's given me the impression that if I don't create something that is up to a teams standard from commit 0

Linus has said that he reserves his abuse for senior people who should know better. It's his antidote to a complacent overclass that mistreats the kernel development process.

I never stated that it was poorly run, acting like this might get better results. I was replying to the poster who postulated what could happen if a hit piece was ran in a public setting against someone for their OSS contributions and how it seemed like reality wasn't already too far off from that to me

Reply on news.ycombinator.com