I've read quite a number of blog posts about scientists who were certain a paper was p-hacked (and proven right later) but who said they would never dare raise their voice publicly because it would kill their career.
I took the side of 781, and he was justin66. My perspective is that you cannot take data, or a concept, that is fundamentally broken, add a little effort, and fix it.
For example, if a collaborator sends data for analysis where the sample size was too small for the effect, or worse, asked to find statistical differences between 1-2 samples per group, you can't fix that with any amount of algorithmic mumbo-jumbo. You can only fix it by redoing the experiment properly. Although we are asked to do this sort of thing almost daily (not always this blatantly).
His perspective is, and maybe I'm being a bit uncharitable, you take what you're given, do your best, clearly describe what you did and honestly note the limitations, and move on. Even if the result is something highly dubious, that is for the reviewers to decide.
But I personally think a little loud whining is called for in the sciences to discourage totally half-baked ideas and data from ever seeing the light of day. But the trouble is the pressures are so intense and funding is so tight, people feel compelled to halfass many things, which is the real problem.
EDIT: I should add I am not talking about data falsification or major ethical breaches like that. I'm talking about things that more amount to a form of pressure-induced willful stupidity, no dishonesty involved. I would speculate this is much more of a cause of the replication crisis than outright dishonesty.
People don't like to outright lie because it could cost them their careers. But they certainly can conveniently forget some basic scientific principles at times if it suits them and they can't get a decent story while being rigorous. And their excuse will be -- well, we clearly documented everything we did in the paper, and the reviewers thought it was OK (because the reviewers do the same sorts of thing). And this, folks, is how you get a lot of shitty papers.
Isn't that still dishonesty though? If you know that you are misapplying statistical techniques to get what you want, how is that different from a pseudo-scientist talking about quantum mumbo-jumbo to get what he wants.
https://rationalwiki.org/wiki/Lying_by_omission
On a bigger scale, this just fuels public anti-science, who is fed up of reading in the news "coffee is good for your heart" and the next day "coffee causes heart attacks". And it's not only because of journalists writing sensational titles, you can actually find meta-analysis pro/against any major food group.
> On a bigger scale, this just fuels public anti-science
Yep... But people have careers to maintain and grants to get.
I totally agree with you and take your side on this issue but the systemic issues are huge and I don't know what exactly it would take to fix them.
I don't disagree.
Au contraire: a few years back my colleagues came to me with an shRNA screen with no replication. I said you can't do this, without an estimate of variability this is meaningless (NOT the first, or last, time i gave this lecture). The reply was, analyze it anyway. I made up some bullshit. The reviewers did nothing. We published in Nature Genetics.
At this point I am cynical enough to believe people include analysts on their papers as an insurance policy to have the "licensed" person to give the seal of approval, and take the blame if necessary, for their shitty data. Which they will never listen to concerns about before publication.
Just today I am working on a qPCR array (yes people still use those) with one "housekeeping" gene replicated 4 times for ddCt normalization. 80 miRs go up, none go down, out of 400. No one sees any problem with this, the data is the data.
Them: Oh, and can you, cgiles, also tell us about the genes regulated by these miRs?
Me: Not really in any principled way, especially when the underlying data is incredibly fishy, and even if I did, we would be saying all the target genes go down. Don't you think people will find that odd? Do you think perhaps there could be a problem with your "housekeeping" gene?
Them: Who cares? Just run the algorithm.
Thank you for this tale. I'm glad I'm not the only one in this situation.
But when push comes to shove, keeping their job and food on the table for their families comes first. So ethical compromises are made.
Similarly, for tech people and privacy. I would wager there are dozens if not hundreds here on HN who have raised concerns with management about the privacy implications of something they're being requested to build. Management says, build it anyway. Programmer reluctantly complies because they like their job.
WRT science it is obvious that it works over the long term. But in the short term it can be quite nasty. So I would say if you are a member of the public, you can safely believe something scientists have said for a long time without controversy, like the helical structure of DNA, but new or controversial findings should be taken with a large grain of salt.