The acceptance of bad scientific practice to the benefit of productivity

I guess it is only a matter of time before the proverbial skyscraper collapses. I have had first hand experience with publishing papers of decreasing relevance. It's almost always due to professors being unwilling to give up their mental model of reality. I had one professor tell me he would not let me publish my result because it contradicted a paper he wrote in the late 70s. It was the grossest violation of the spirit and purpose of science I've encountered, and it quickly turned me off from the whole institution. Egos in academia are often unchecked.

That doesn't sound SO bad. Selection criteria seems like a major issue here, and the author only talks about the validity of the results.

Why do we care about physics? Or psychology? There are answers to these simple questions, but I don't think the answers inform the selection criteria. We don't care about physics because academic researchers need to publish papers or get funding.

Having any old non-expert on the panel could be problematic. Especially if they become political tools, and right now they'd definitely become political tools. But it would seem reasonable to me if there were adjacent experts. What do engineers or chemists want to see in physics? These aren't hostile interlopers. But if nobody outside a field cares about a line of research, or trusts a line of research, then maybe that's a bad line of research.

> For starters, every scientist should know how being part of a group can affect their opinion. Grants should not be awarded based on popularity. Researchers who leave fields of declining promise need encouragement, not punishment because their productivity may dwindle while they retrain. And we should generally require scientists to name both advantages and shortcomings of their hypotheses.

The first and last points look good to me. The middle two, not so much. They certainly sound reasonable, but what does a grant system not based on "popularity" mean? How is it different from what we have now, where we find some (putative) experts in the area, put it in front of them, and ask what they think? We need some way of deciding what to fund and what not to fund, and who exactly is better equipped to make those decisions than existing researchers?

Similarly, I like the idea that people should be able to switch fields. But the implicit argument seems to be that the emphasis on publications makes this difficult, so publications should be de-emphasized in assessing researcher quality. But what is better? Asking a researcher's peers? Then it is a popularity contest again. Weighting publications differently, i.e. "what are your 5 best papers?" rather than "how many papers do you have?"? Then who judges the 5?

For all the criticism heaped upon bibliometrics: they get used because assessing the quality of most research is difficult, so even noisy filters like conferences (in CS) or journals are taken as a useful signal. It's not clear to me what better options exist, especially when trying to choose between many options. It's just not possible for one person or even a small group of people to truly familiarize themselves with the bodies of work of all 100 candidates for a faculty position/grant/fellowship/etc.

"Or look at Brian Wansink, the Cornell Professor with the bottomless soup bowl experiment. He recently drew unwanted attention to himself with a blogpost in which he advised a student to try harder getting results out of data because it “cost us a lot of time and our own money to collect.” Had Wansink been aware that massaging data until it delivers is not sound statistical procedure, he’d probably not have blogged about it."

Physics is one of the least susceptible to this issue, but no field is immune. The issue is that 'science' has become more about publishing than actual discovery. The two should be synonymous, but they are not.

[1] - https://en.wikipedia.org/wiki/Educational_attainment_in_the_...