Bad science persists because poor methods are rewarded

It's hard to find information on what the problem was with that one, so here's a link to an article from Slate:

http://www.slate.com/articles/health_and_science/science/201...

Yea, because it is really all the overworked and underpaid graduate students faults... I have never met a single graduate student who rushed to publish shoddy work on his/her own. However, I have watched time and again while young impressionable graduate students are slowly but surely guided to publish shoddy work by overworked and desperate professors who feel they have no choice but to keep getting "academic currency" (e.g. publications).

Thats the only time when things have a general interest. Negative results are very important to people directly involved in studying something but what general interest (or value) is there in a story "chemical xyz123 does not work as an effective catalyst for the reaction between abc123 and def678" for anyone not directly researching those things?

It is actually a huge difference. Raising kids, for instance, it isn't enough to just reward good behavior. You have to actively and fairly penalize bad behavior. And if you are setting up a losing game or your kids, expect bad behavior.

Which is what most of this comes down to. For most participants, they are in a losing game. The absolute best they can do is get through with their project and move to the next one. About the worst they can do is spend more time on what they are currently doing.

So, I don't want my comment to be a "screw this article, they got it wrong." No, contrarily, this is a discussion that needs to be had. I personally think it is not enough to just look at the rewards people set themselves up for on the bad behavior. You have to look at the penalties they have for "good" behavior.

This is not new. Few years ago the lament was we don't teach how to fail. That is all I am echoing. Failure is heavily punished. Maybe it shouldn't be.

I disagree completely. Ackowledging that "good methods are penalised" would challenge the line generally taken by the Economist in its socio-political pieces: that if classical liberal economic models were applied to everything, everything would work better.

What's being described here is the consequence of the application of "business thinking" and market-driven funding of science, with all the unintended consequences of trying to apply quarterly metrics to pure research. Suggesting the problem is "not enough punishment" avoids having to ask philosophically hard questions.

I argue that you are modeling out real expense. In this case, the cost of not publishing significant findings.

You could try to penalize bad faith actors. You could also try to stop putting undue pressure on good faith actors.

It isn't just spending money. But opportunity cost of throw away work.

As it is we have what can be modeled akin to baseball where most at bats are failures, but not even swinging is frowned upon and eventually leads to you not being sent to bat.

Given that plan to decimate the economy of America's 4th largest and most racially diverse city (Houston, TX), I am quite keen to know if these issues also affect climate science.

Don't get me wrong, I'm still pretty convinced that anthropogenic global warming is happening. But if there is a change that it isn't and we can avoid all of the murders and suicides that come with throwing large numbers of people in one area out of work, we should investigate the possibility.

Nope. It just means that you have to come up with a shittier null hypothesis. Of course, it still has to get past peer review, but this can often be accomplished by complication and obfuscation, which have the side benefit of giving you plausible deniability in case you are discovered. Sufficiently advanced cluelessness is indistinguishable from malice, and science, in its present form, rewards them both.

In a lot of fields you can. In fact an entire field (ego depletion) seems to have been ENTIRELY produced by this type of publication selection bias.

It's a society-wide problem that is caused by competition and status seeking. Pretty much everything is optimizing for "winning", whatever that means, instead of doing things correctly, conscientiously, and carefully. The latter will often make you "lose", and there's no system rewarding you for it besides an internal one. Then there's nothing left for people to do but try to "win", because resources are scarce. The more you optimize for "winning", the more nasty your methods get, because there's always something you can sacrifice, until nothing is left.

Another unseen side effect of this is that a lot of people see this, combined with the low returns from science, and decide to go do something else, because if they are going to put up with the garbage they can as well earn more money doing it. And some among them then realize that they can have a greater effect by generating billions of dollars and then using them on whatever they want, instead of wasting time in a lab and asking for scraps from the government.

Untangling this requires a complete mentality shift.

Yes but why are scientists in the position of selling?

Why is that their position in whatever this negotiation is?

First I should explain I believe Science to be a special type of Education, and that by addressing the specific here on HN we are not attacking the general problem.

There is nothing complex about this subject. It is just bad governance leading to institutional weakness. Let us look at long term trends, since that is what ultimately matters.

In 1900 to a first approximation there existed 1000 physicsts on earth. A century later that number is larger by a factor of (at least) 1000x.

Yet I wouldn't claim physics has produced 10x the results, let alone 1000. This requires explanation.

I am not ganging up on physics, the problem is across all the sciences.

This means either of two possibles.

1. There exists diminishing returns even in the discovery of new ideas themselves. We should be seeing an explosion of new fields of science, for every 1 field in 1900 we should see 100 new ones today but we do not.

2. Science as an institution has gradually succumbed to some kind of rot from within.

I submit to you it looks like the first but is actually the second proposition which is the truth.

Now I will come clear. It is democracy that has accomplished this. This is because public money must be tethered to some kind of measurable outcome, an easily understood metric, for future flows of monetary assistance.

The causality goes like this.

Public finances must be justified in the development of Education and Science. Bureaucrats are in essence the farmers of students or scientists in a similar way to cash crops or Prussian forestry management. More units, more volume equal more success. Thus, the limits on the network of bureaucracy are artificial limits on the potential for growth.

Going 'wild' is not tolerated even if it would lead to more interesting outcomes (which is what we want!). It is seen as expensive when it is nothing of the sort.

The resultant output is the mass production of mediocrity.

The larger the apparatus, the more all encompassing it becomes until its standards and objectives become universal. Its gravity even influences the trajectory of systems outside of it e.g. privately funded research due to its incredible size (because those privately funded researchers were inoculated into conformity with a public system or a private system which a curriculum based on the public model).

All that I have said should be testable.

Suppose you have the following arrangement.

1. Students or Scientists with independent source of income. We may have a filter for motivation but that's about all.

2. Educational assistance which is funded through present and former students/scientists.

3. There does not exist a standardized curriculum or set of objectives. It is all blue sky. Any resultant consensus on common topic areas is by choice. Interdisciplinary activity would explode because the subjects are not focused on a 'path', which necessarily distorts and cuts down on the number of potential possibilities.

Then I am convinced you shall see a thousand flowers bloom.

Edit: The more 'elite' a university is, the more it resembles the arrangement I describe. It is undeniable that the best parts of the Educational system are the most aristocratic, while the worst are the most demotic.

Same question. I would assume this is partially based on replicability, or doing the study over and getting the same results. Recently, there was an attempt to replicate major studies in psychology and only about 36% were replicated. I think something similar was in medicine.

I've seen copious bad papers in EE and MechE journals. I'm not qualified to evaluate chem or physics research, but you'd think if any field would be hard to fudge, it would be engineering. Turns out this is not the case. The bad engineering papers I've seen tend to draw overly-broad conclusions while failing to document the methods adequately.

> Changing the descriptor would be a classic case of bike-shedding.

Strongly disagree. Things that are "scientific" get way more acceptance by the public then things that are "hey, that's just, like, your opinion, man."

For instance, in what sense at all is "political science" an actual science in the sense of chemistry or physics? It seems to me that it's an entirely different thing, and so I'm not surprised at all when a political science "result" fails to replicate or is outright bullshit.

Science has come to mean, in many cases, "area you can get a PhD in" vs. what we tend to act like it means, which is "area where the scientific method applies." I guess I'd like to see the word science only used for the latter meaning if we're going to continue to give "scientific" disciplines extra respect.

If instead we want to continue to use "science" to denote "anything you can get a PhD in", then we should seriously degrade or eliminate altogether our instinct to trust "scientific" results.

> 1. Real science that demonstrably works now.

One of the things I noticed in college is that the classes I took in this category never actually taught "the scientific method."

It was only ever in the so-called "soft" sciences that we got the scientific method lecture about creating a hypothesis, doing some tests to verify, checking results, etc.—usually in the first week.

So maybe a way to identify classes still in your category "2" is to see if they explicitly teach the scientific method or not. If they do? Category 2. If they're obviously scientific and don't teach it? Category 1.

(I've also noticed that an easy way to identify a crackpot on the Internet is to see if they trot out the scientific method discussion in the first 1000 words. If they do? Crackpot.)