Science, Now Under Scrutiny Itself
nytimes.com
nytimes.com
Recently the depth of negative influence, corrupt findings, blind faith and religious-style zeal within the scientific community and pseudo scientific communities (like Reddit) has become absurd and embarrassing.
Science is a method, not a conclusion. I hope it becomes 'open', and fast.
The religious zeal I think comes from fan boys/girls not from scientists themselves.
The increase in retractions can mean either that (1) there's more bad science now than there was before, or (2) we've got better at spotting bad science. Without further evidence, it's difficult to tell which one is the case. Perhaps it's a mix of both.
Scientist could have repos with full version control, so you could go back and look at their experiments from day one. All their data would be accessible, as well as their statistical analysis, and you could see the evolution of their written paper.
The repos could start off private, and then opened on the day of publishing, so others couldn't take credit for their work.
Just a thought.
> The repos could start off private, and then opened on the day of publishing, so others couldn't take credit for their work.
Funny you should say that...
I work at the Center for Open Science, and our founder was quoted in the article. One of the major use cases for our primary product (the Open Science Framework) is to facilitate preregistration.
Here's the relevant section of our "Getting Started" page, detailing how preregistration works: https://osf.io/getting-started/#registrations
You can find out more about COS at http://cos.io.
One of my first OOP projects trying to model the real world in hierarchical classes. It was a good non-contrived example of a need for multiple orthogonal inheritance methods as any one can easily be forced into contortions in modeling even simple structures.
I tried to write a project with my own trig implementations, just because. That'd at least be funny...
But the rest are a few lines of drunk-coding, or something I found a library to do and never developed, or in some other way a waste of time to look at.
So I don't want to hold scientists to standards I wouldn't want. Nor do I want to miss the value in the noise.
So maybe we should have a honor-system somewhat like this - when you first describe to project to anyone looking for funding, or discussed it with more than a handful of peers, or ... then you make a note in your lab book. "Research direction: see if flatworms are really flat because Y - hope it'll give insight into Z". And your internal audit system submits all of these to a pool of peers who pick by interest in the area. If it's chosen, an auditor checks if the idea was ever really developed, dropping it if not, and summarizing whatever is there (or letting you write a paper) if there are results.
It sounds like a lot of work, but considering that everyone is asking for grants and being audited anyways, and this would replace vast swaths of less-pointed simple bookkeeping audit with peer-review of your methods, etc. In other words, likely higher quality results for the funding bodies in less time, and with the additional benefit of being able to publish interesting failure results.
Thus achieving a balance between researcher privacy and ability to try without public mockery on failure, funding efficiency, and scientific benefit.
The problem with these lab books is that they're often the personal property of the scientists and, as such, almost a sort of diary. They're written with yourself as the intended audience and can sometimes be impenetrable for anyone else. If you're working in a commercial lab, as opposed to research in academia, these books might be considered the property of your employer. In this situation you might try to make them more friendly to other human beings, but that often isn't a priority.
Putting things into a digital form would, in general, take a lot longer. Even with good proficiency in LaTeX, complex equations are quicker to just write on paper. Hand-drawn diagrams are also very common in lab books, and the software/hardware to draw easily in a digital form is not yet ubiquitous. Finally, digital storage occasionally dies. A lab book is a precious possession whose loss can cause months or years of difficulty. Physical hard-copy is easier to store long-term. Some people have undergraduate lab books from half a century ago sitting on their office bookshelves.
There is nothing that is technologically insurmountable to moving to a digital lab book format. Microsoft's surface tablets make digital drawing pretty painless. Cloud storage could be maintained long-term. Etc. The benefits of digital lab books could be considerable. They'd be easier to search and distribute. You could include copies in a digital appendix to an experiment (currently, most journals discourage large attachments because they don't like to pay for storage). To my knowledge, nobody has actually put everything that's needed together into a single cohesive program. You'd want a note-taking program with full LaTeX and freehand drawing support. You'd need to be able to paste in tables, graphs, and data files easily. It would need to have a long-term archiving solution and a really good interface. It would need to be as fast or faster than writing on paper for pretty much everything. It would also probably need to be open source, as scientists would probably not trust proprietary software that might make their notes difficult to access at some point in the future. Finally, the hardware to run it on would need to be ubiquitous.
This is also an issue where investigators apply custom software to complex systems. Getting source code can be nontrivial, and documentation is often inadequate. And there may be bugs that were avoided rather than patched.
Further complicating things, individual researchers don't actually own the books. Often, an institution or the lab owns the rights to the book. This can make individuals who are otherwise keen on "open science" hesitant to place copies of this information in a publicly accessible place.
See page 4 of this manual from the NIH (National Institutes of Health) about the ownership of notebooks.
https://www.training.nih.gov/assets/Lab_Notebook_508_%28new%...
Science, like every human institution, is flawed. That's not a reason to reject everything about it.
Over time people can't help but depend on science's results, and new generations take it to heart. It's actually similar to how people can't help but depend on open source tools/platforms, as much as they hated "free-tards" and said "you get what you pay for" in the past.
So it's pointless to try to rebuke the "scientists always end up being wrong" like of argument. If anything, I think scientists should just bite the bullet. Theory proven wrong? That's proof that science works!
Until it's been reproduced and validated by a few outside - and hopefully disinterested - third parties, we shouldn't accept it as fact but as a theory that could be replaced.
Confusing peer review with fact is a public-perception problem, and a lot of that comes down to various antiscience groups whose work isn't even peer reviewed, let alone tested and challenged.
Just a Google search can reveal plagiarism (by accident even).
A bot can find issues with made up numbers (Benford's law for starters)
It's all pretty cool stuff for science.
> But it is sometimes much easier not to replicate than to replicate studies
I wonder in what cases its easier to replicate studies. Probably when the study involves doing nothing...
For example if you create a natural language parser that you claim is more accurate than another, you may need to replicate the other in order to compare accuracies. Sometimes you can just take the authors' reported accuracy, but often it's not that easy (maybe they just tested on English and you're interested in performance in other languages, maybe you want a different metric, maybe you want to compare on some standard set of features, etc.)
And the push for open data repositories and open science to increase accountability has to be a good thing.
I've read that the journals realized that they had a captive audience, i.e., that college libraries had no choice but to subscribe in order to support their faculty, and the publishers raised prices dramatically. There was a backlash from some schools at the time.
Here's a paper that explores the idea and its impacts - http://www.vanderbilt.edu/econ/faculty/Wooders/APET/Pet2004/...
Ah, the irony of pluck-a-number arguments when discussing this particular topic...
Why is this a problem? If the experiment's design is not in conflict with the new findings, why complain?
The problem is we're not matching up our statistical analysis to our testing when we look for findings then naively test hypotheses based on those very findings.
Statistics can be true and false at the same time. Depending on interpretation of the data and how you try and 'spin it'.
This is why I'm wary of any study that only shows their statistics, but does not share their testing methods.
Bayes can be hacked by ommitting info, but not by things like this, I believe.
The inference in Bayes only depends on the data, and the flips of other coins doesn't make a difference if independent. Frequentist testing can depend on things like stopping rules and hypothesis tested, which aren't correlated to the actual truth and therefore should have zero effect on inference.
They can cherry pick and only show some flips of that coin, but then they really need to be outright lying or you'll ask why only some flips were reported.
Two-up ("Come in, spinner!") is a coin-flipping gambling game where people bet on whether a punter will come up double-heads or double-tails. The house makes it's money when neither come up five times in a row (with some variations).
This is the reason experiment repeatability should be considered of crirical importance especially for 'gooey' disciplines like medical and psychological research where there is little more than statistics to go with. Was the experiment just mininf random noise for correlation - 'simple', just repeat the experiment. If results can't be repeated it's more likely the original authors just botched the experiment, massaged their data to look how they wanted or just got lucky with a random number sequence.
This is similar to the sharpshooter fallacy. If I shoot at the side of the barn then draw a bullseye wherever I hit I can pretend to be a good shot.
This generates an unrealistically high statistical significance (unrealististically low p-value) because the question should not be, "What are the odds these two things are correlated by chance" but "What are the odds that any of of these things is correlated by chance with this other thing?"
The significance of the result is far below the reported p-value from an analysis that assumes only one thing was investigated. You frequently see incorrect p-values of this kind even when the researchers acknowledge they tested for multiple things.