Scientific Regress

I am a post-doctoral research scientist that has wanted to be a scientist since early childhood (6-7 years old). The structure of how science is practiced seems to have changed substantially in that time frame, not for the better. Some of my childhood naivete plays a role in those perceptions, but many older scientists share that perspective.

A lot of the problems in modern science revolve around the growing focus on bringing large quantities of research funds to your institution (which gets a large cut). Getting grants has an increasingly large need for political clout due to peer-review and increased competition due to reduced funding. Consequently, labs are getting larger and more hierarchical (i.e., an increased number of long-term post-docs and research scientists). As the organizational structure of science becomes less flat, the influence of those at the top is just reinforced and tends to constrain dialogue to fit established ideas. I think that the long-term progress of science depends on placing small bets on a greater variety of ideas rather than doubling down on fewer. Unfortunately, it will always be perceived to be safer to fund conventional ideas. Peer review enforces short-term, safe bet approaches.

Large labs can churn out lots of papers, even if they are relatively financially inefficient. One way to correct this (if you agree that is a problem) might be to normalize grant scores by previous grant funding to the PI. i.e., (X papers of Y impact)/Z dollars of funding over the past 10 years. Double-blind review of grants might also help, but blinding is relatively easily circumvented.

>But it adds to this a pinch of glib frivolity and a dash of unembarrassed ignorance. Its rhetorical tics include a forced enthusiasm (a search on Twitter for the hashtag “#sciencedancing” speaks volumes) and a penchant for profanity. Here in Silicon Valley, one can scarcely go a day without seeing a t-shirt reading “Science: It works, b—es!” The hero of the recent popular movie The Martian boasts that he will “science the sh— out of” a situation. One of the largest groups on Facebook is titled “I f—ing love Science!” (a name which, combined with the group’s penchant for posting scarcely any actual scientific material but a lot of pictures of natural phenomena, has prompted more than one actual scientist of my acquaintance to mutter under her breath, “What you truly love is pictures”). Some of the Cult’s leaders like to play dress-up as scientists—Bill Nye and Neil deGrasse Tyson are two particularly prominent examples— but hardly any of them have contributed any research results of note. Rather, Cult leadership trends heavily in the direction of educators, popularizers, and journalists.

As much as I think the essay is great, this paragraph is a terrible ad-hominem argument for the existence of a "cult of science" or why it is dangerous. It doesn't matter if people wear xkcd shirts, Matt Damon uses profanity with "science", or if people like a facebook page. Laypeople having fun with "science" does not make a cult. A better argument would be to show how people cargo-cult science, and why it causes problems with science. The author fails to do both.

IFL is dangerous not because it is only pictures of scientific phenomena, but that bullshit is constantly posted on the page to be consumed by laypeople. The cult of science is dangerous because it has failed to teach people how to properly evaluate claims, and believe anything with the word "study" in it. Woo-pushers have appropriated the vocabulary of science indistinguishable to a layperson. Andrew Wakefield resurrected long dead pandemics by falsely publishing in the Lancet. This is what makes the cult of science dangerous, not the word "bitches".

My father spent every night of his 30 year career researching semiconductors with a deep belief he would be fired next week if he couldn't bring in enough grant money. He is an amazing scientist and you wouldn't be on the internet today without his spadework but he couldn't give a rat's ass about funding.

One day a former grad student showed him some pictures.

"That's my lab!" He said.

"No, that's the replica I built of your lab in China for 1/100 of the cost."

Science is such a noble pursuit. If only it were separable from humanity's endless supply of greedy pricks.

While these problems clearly exist, I would argue that we're still making more progress than "regress". The whole premise of the article seems to be that the "good old days" are gone, but I'm not convinced that things were much better then, after all, human psychology hasn't changed. For example, psychological bias can be found even in physics, as described by Feynman:

"We have learned a lot from experience about how to handle some of the ways we fool ourselves. One example: Millikan measured the charge on an electron by an experiment with falling oil drops, and got an answer which we now know not to be quite right. It's a little bit off because he had the incorrect value for the viscosity of air. It's interesting to look at the history of measurements of the charge of an electron, after Millikan. If you plot them as a function of time, you find that one is a little bit bigger than Millikan's, and the next one's a little bit bigger than that, and the next one's a little bit bigger than that, until finally they settle down to a number which is higher.

Why didn't they discover the new number was higher right away? It's a thing that scientists are ashamed of—this history—because it's apparent that people did things like this: When they got a number that was too high above Millikan's, they thought something must be wrong—and they would look for and find a reason why something might be wrong. When they got a number close to Millikan's value they didn't look so hard. And so they eliminated the numbers that were too far off, and did other things like that..."

>>>First Things is published by The Institute on Religion and Public Life, an interreligious, nonpartisan research and education institute whose purpose is to advance a religiously informed public philosophy for the ordering of society.

I'm not quite sure what to think about the bias we might be seeing in this post from this site. This is quite a long essay and there's not a single link on a site that seems like it would be likely to be quite biased. I'm not saying anything is wrong with this, but I guess just consider the source? It could very well be fine.

Many years ago I started out as a Chem major. After 3 years and many hours I quit.

My grades were swell, I still loved science - but I couldn't be honest with myself that I was going to be a real scientists in the end.

So here's what typically goes wrong: You get a lab assignment. Somewhere along 4th hour or second week you screw up and grab the lab ass for help. He says "sorry - just keep going."

You won't get a do-over. You can't afford to start over because the expected competence requires immediate good results.

Your lab time is limited and resources are scarce. If you want the grade you'd better "learn from your mistakes" and "be more careful next time."

What about the results? Well you already have a pre-conceived notion of what they should be. Maybe you get them from your mates, or look them up.

Learning from our mistakes is "science code language" for pushing small known nudges and data "massaging" as an acceptable method for passing the course.

Perhaps at one time students really did learn and grow from these mistakes - but the modern concept of failing is simply a quick exit from a highly competitive major.

All the while my own scientific rigor which I was supposed to be enforcing on my own results was slowly corrupted.

I couldn't truly say that my practise in the scientific method was honestly the truthful result of my own observations and methods. Perhaps I was just too anal at the time - so be it.

I worked in an environmental testing lab and the sloppy procedures practiced were sometimes much the same.

Circumstantial? Yes. Take of it what you will. But if my experience is like that of others - such corruption as reported here doesn't surprise me.

Science should be about failure as much as it is about success. Failure is a valid result - but we often fail to oblige real and honest failure as scientifically and (most importantly) educationally valid.

That was my draw to CompSci. Our very embrace of failure as a tool for learning. I absolutely love it.

Certainly, the reproducibility crisis in social science and cancer biology is important, and deserves attention. But this article mostly strikes me as profoundly overstated. Yes, science is done by humans, and is thus imperfect. Yes, science's approach to truth is nonmonotonic. (I think most working scientists would consider this obvious.) Yes, scientists have various incentives to hype surprising results, cut corners, and even cheat. Yes, a lot of bad science gets published. But clearly science is making progress, since we clearly know more than we did 5, 10, 20, or 50 years ago. Are there things that could be done to improve the way science is conducted? Almost certainly (e.g. preregistering experimental trials, requiring power analysis in published papers), but this article is remarkable quiet about such things. In the end, what is author even really saying? That lay people should trust scientists less than they do? That doesn't seem correct to me. For instance, on climate change, it seems that normal people trust scientists a good deal less than they should, instead placing their trust in talking heads, politicians, and people who are unambiguously paid by oil companies to advocate on their behalf. So: yes, science as practiced in 2016 is imperfect. But what exactly do you propose replacing it with?

I agree that there are real structural issues, especially in certain fields, and especially involving the incentives for funding and publishing. However, I think that the overall impression gives some mis-characterization, at least from my perspective as a molecular biologist / geneticist.

First, I think that the most prominent studies are the most likely to have issues; generally they're doing something new, often with new methodology. It's here where extrapolations tend to be made that are the most dangerous. It's also not surprising that social sciences have such issues; they don't have rigorous tools (i.e. genetics) that can effectively ground their work. It makes compounding issues much harder to catch.

Related is the re-testing issue; in many fields, subsequent work will catch errors in previous work. If you work with a mutant and then someone else works with it, they'll see if it behaves differently than expected from the previous work. Germplasm travels, and it's the ultimate arbiter of truth. This usually does lead to further scrutiny and fixing the issue. The real problem that I've observed isn't that the errors aren't caught, but that a formal retraction isn't always done. Sometimes it just gets contradicted in a subsequent paper (and often with some relish) without ever resulting in a retraction. The editors of that journal clearly have a responsibility here that they are failing to uphold.

However, despite these faults, it's quite clear to those of us with 'boots on the ground' that you can't hide from the data; as long as you're using solid genetics and doing 'real' experiments (e.g. western blots, in situ/immuno-localization, simple gels/pcr, etc.) you can only hide for so long. The exception is if no one keeps working on it, in which case it's probably not that interesting to begin with. This also leads to a deep suspicion of bioinformatics among geneticists because we see how often things go wrong, and what is needed to make it right. Fortunately, genetics can still be used to great effect.

Ultimately, I'm not suspicious of the large body of work in my field. Most of it is based on extremely solid forward genetics that withstands lots of testing. Even now, great value is placed on these 'old school' methods because of how robust they are known to be.

This article appears to be bent on buttressing an anti-scientific religious viewpoint rather than improving science. (Read the last few paragraphs.) The conclusion strikes the same themes that I've seen many times in the religious anti-science movement: science is a religion, a cult, etc.

Sure, statistics are difficult and can lead to incorrect conclusions, but that's why we make sure that a scientific claim is falsifiable. The fact that we can test the claims are where much of the power lies. Let's not forget that a few centuries of the scientific method have made human lives so much better than millennia of religion.

Strange to see that HN is supporting the views of anti-science propaganda, promoted by religious groups. The great thing about science is that it is falsifiable. So, yes, mistakes will be made, specially with so much pressure to publish more papers. But finding the mistakes in these papers is another incentive that leads science to progress, while religion will forever keep pointing fingers without anything better to offer.

I think this is largely a case of 'moloch' as described by Scott Alexander: http://slatestarcodex.com/2014/07/30/meditations-on-moloch/

In that long (but excellent essay) Scott points out that self organizing systems like human society can sometimes arrange themselves in horrible local minimas which can be very difficult to escape.

I think the way modern science is organized is decidedly suboptimal. Note that I'm talking purely sociologically: science, as a method, is still by far the best thing we've developed to understand the natural world. However, the incentives around publishing / grants / hiring work are broken in a myriad of ways, from small to massive.

For example:

- High prestige journals/conferences are basically a crapshoot: http://blog.mrtz.org/2014/12/15/the-nips-experiment.html - consider that getting a paper accepted in NIPS/published in Nature might completely change the way your career shapes up.

- Grant funding is incredibly competitive, and the way they are awarded is also a crapshot: https://psmag.com/why-the-national-institutes-of-health-shou...

- The pressure to publish/obtain grants drives people to either make up data to obtain fancy publications, or even to kill themselves: http://www.dcscience.net/2014/12/01/publish-and-perish-at-im...

- We reward researchers for brilliant clean discoveries, not brilliant methodologies. However, the outcome of a serious research project is the one thing that a scientist cannot control: science investigates the unknown, if someone explores a plausible hypothesis in a clever way, and the hypothesis turns out to be wrong, the scientist didn't do anything wrong.

- PhDs are often used as cheap labour. A very famous PI once bragged that before they got their new fancy robot, they just had 10 chinese postdocs handle all the plating.

- Private companies make an insane amount of money from publishing scientific research that's carried out with public money, and that's reviewed by scientists (largely) paid with public money who donate their money for free.

I could go on for ages - and so could almost every researcher/ex-researcher on HN. There's a few no brainer changes - but the problem is that almost everyone in power who could affect change stands to benefit from the current system.

Great article. I would say this is inaccurate though:

>"What it really means is that for each of the countless false hypo­theses that are contemplated by researchers, we accept a 5 percent chance that it will be falsely counted as true—a decision with a considerably more deleterious effect on the proportion of correct studies."

The hypotheses that the statistical "hypothesis testing" framework is usually applied to often amount to "two groups of cells/animals/people are samples from exactly the same population". Then there will be assumptions about the distribution of this population, etc. As has been noted by many (see eg Meehl 1967), such a hypothesis is pretty much always false and nothing but a strawman.

http://www.psych.umn.edu/people/meehlp/WebNEW/PUBLICATIONS/0...

The problem is that fields don't advance in a continuous process, but its in bursts of stop and go. This state with a lack of progress and entrenched thought schools is prelude to another fracking of the unknown. Somebody in all this science looks on the "outliers" and will connect the dots. And the truth is, that not the science community grants the ultimate reward- but society and economics, via application.

You can assemble a thousand followers, claiming that electrons are little golden dwarfs, running through the metal if fed with potatoes, but the world wants chips without chips, so what hinders science? Globalization, border-lessness, in a ironic twist, cause where can that industrial revolution flourish and reward overcome all prejudices? When the potato religion/ideology is the same everywhere, there is noway to run too for the first battery maker.

One of the charlatans is not one. Better to feed a hundred of them, for the one to prevail.

This article makes some good points, although ones that have been made a lot recently, but it hugely oversells the regress argument.

If that was happening, it should be quite easy to measure in some areas: 5 year cancer survival rates should fall, corn yields should drop, and so on. For the most part, we don't seem to see that.

If it was all some sort of faith, and the bad findings were accepted, we should see more-or-less random changes instead of steady improvement

TLDR: some scientists are doing science by critiquing what some other scientists did, using statistics. The authors are a wee bit confused about the fact that science is needed to weed out the 'bad' science from the 'good' science.

This is an obviously biaised text by an author with a fundamentalist religious agenda.

He cites all well-known and not at all surprising problems with peer review, which means that a sizeable proportion of published science is wrong -- wrong as in somewhat innaccurate and likely perfectible, and in now way as wrong as only religious ideas can be, i.e. completely false and unfounded. Asimov wrote a great essay about this Relativity of Wrong [1].

Indeed, that fact that science isn't regressive, and our knowledge is getting more accurate with each passing day, is self-evident. Suffice to compare what we have now with what we had 20 years ago, and its clear that we are not regressing.

Now, as to the specific examples that are falsely used to proove how wrong this wicked science can get:

1. "one hundred published psychology experiments".

Seriously, anyone in the field knows that these studies are rubbish, and all correlations are obtained after running so many statstical tests that they are clearly due to chance.

2. "half of all academic biomedical research will ultimately prove false".

Again, this is obvious. Many initially positive findings will be due to chance, especially when we are talking of small, exploration studies. Yes, many initially promising molecules did not live out to the hype, but others did, and spectacularly so. For instance, progress in oncology has been amazing lately. People who had life expectancies of mere weeks in early 2000s now routinely survive for years with their metastatic cancers.

3. "but are unlikely to mention a similar experiment conducted on reviewers of the prestigious British Medical Journal." [as opposed to Sokal's Social Text hoax].

Apples and oranges. Sokal's article is plain old gibberish, whereas the experiments in medical litterature were to assess the reviewers capacity to assess for methodology failures of otherwise plausible studies. Most of these methodology failures are weeded out by dedicated statistical and clerical staff.

Indeed, I would argue that some of those methodological problems were pretty minor, e.g. :

"Poor justification for conducting the study; No ethics committee approval; Failure to spot word reversal in text leading to wrong interpretation of results; No explanations for ineligible or non-randomized cases; Safety. No mention was made of monitoring patients for untoward effects; Format. The abstract was not written in the structured form requested by Annals; References. No reference cited was more current than 1989 despite the fact that there were numerous more recent studies; 5.Presentation. There were multiple grammatical and spelling mistakes, including the misspelling of propranolol as “propanalol." [2,3]

[1] http://hermiene.net/essays-trans/relativity_of_wrong.html

[2] http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2586872/

[3] http://www.annemergmed.com/article/S0196-0644(98)70006-X/ful...

disturbing.