We're so good at medical studies that most of them are wrong
arstechnica.com
arstechnica.com
This would be a welcome change. It seems very strange to me that it is not the accepted practice to make available any source code used in the analysis. It's an essential part of their methodology, and any bugs in that code could produce hard-to-detect flaws in their results (dropped data, rounding errors, etc). Just look at the work jgrahamc did on climate change analysis: http://news.ycombinator.com/item?id=1128782.
In human genetics we have Mark Daly, David Altshuler, and Eric Lander to thank for doing the thoughtful theoretical work beforehand to identify the alpha for identifying genomewide significance for any study: 5E-8. To my knowledge, GWAS results are much less commonly found to be invalid in a later study.
The message is this: everyone knows at least one method of correction for multiple testing, and virtually everyone knows that they should be doing this. No journal should publish papers that merely show nominal, instead of corrected, P values.
Two experimenters contrast treatments A and B. They both have A preferred to B in first 5 patients, and B preferred in the 6th. First experimenter planned to run 6 experiments and count the number of successes, so they get P-value 0.11 for the hypothesis that A is better. The second experimenter planned to run comparisons until B is preferred, up to 6, and got P-value of 0.03. (see Appendix A of http://www.annals.org/content/130/12/995.full.pdf+html)
A realistic example of this issue coming up http://www.jstor.org/pss/2336980
Another example is choosing between one-tailed and two-tailed t-test. When you ask for the probability of effect being as extreme as observed x under null hypothesis, should you ask for probability of effect>x or |effect|>|x|?
Eliezer Yudkowsky goes into some discussion on this http://lesswrong.com/lw/1gc/frequentist_statistics_are_frequ...
The most illustrative example of subjectivity of hypothesis testing is probably the issue of testing strings for randomness. There are many tests for testing whether a particular string of bits is generated by a Bernoulli process, with not one having a legitimate claim to being "the right one"
One way to remove the bias is three-way triple-blind testing, ie to measure effects of existing, new, and placebo treatments, have statistician analyze datasets while blinded to their true labels.
reviewers needed to hold studies to a minimal standard of biological plausibility
There's two "good" ways to do this (as far as I can see). 1) Come up with a biologically plausible idea and test it, using statistics to look at results. 2) Find a pattern in the statistics and find a biologically plausible explanation.
The biology alone isn't enough, you need to statistics to back up and show actual results. However, using statistics alone and in the way indicated in the article (looking at every test and every subgroup, etc...) is exactly what your saying: avoiding rigorous thinking in favor of getting a result.
If you can handle calculus, which most scientists take, then you can handle likelihood ratios, believe me.
But the incentives are terrible, which is quite a different thing from supposing that the average PhD is too dumb to learn good statistics if the incentives were strong.
http://www.plosmedicine.org/article/info:doi/10.1371/journal...
The other problem is that mainstream journalism does an abyssmal job of reporting science. How many times have your read a newspaper article or seen a segment on the evening news that tells you that X is good for your health or X is bad for your health as if it was an absolute truth, a law of nature on par with V=IR or F=ma? Then, out of curiosity, you look up the actual journal article published by the scientists and find their claims to be considerably more modest.
You do point out a very important phenomenon, however. A good many doctors don't know a thing about proper statistical analysis, or bayesian reasoning. They implicitly trust whatever research falls into their laps, even if they don't fully understand it, and they begin to treat based upon their faulty understanding.
The way he sees it, nutrition is a separate specialty done by people who study less than doctors and get paid less than doctors, and it isn't taught in medical school, so he isn't embarrassed at all by his ignorance. He regularly gives advice to people who have risk factors like high blood pressure or high cholesterol, but the advice he gives them (eat in moderation, eat more vegetables and less junk food, exercise a little) is pretty generic, and none of them follow it anyway.
Doesn't that kind of justify his ignorance, though? Doctors don't have time to be experts on everything (or on anything, really), and it seems unlikely that an ability to hold informed discourse on the nutritional value of broccoli would have anything more than a marginal effect on outcomes for his patients.
This is a waste, because the people who are in the best position to understand and evaluate information about living a healthy lifestyle have no influence on the production or consumption of this information, except when they themselves get in on the business.
But the generic advice you described is perfectly adequate for these people 90% of the time.
But my point was that very few people even bother asking him for guidance. It's silly that educated people just assume that by reading a little in their spare time they can acquire a better practical knowledge of nutrition than a professional with four or more years of intense training in human medicine. It's mind-boggling that they're right. For instance, if my friend could name a couple of high-calcium non-dairy foods -- which I'm not confident he could -- it's only because he read some diet books looking for ways to lose weight. He could probably name more symptoms of Chagas disease than dietary sources of vitamin D.
Maybe if doctors were paid to keep their patient base healthy rather than just fix them when they're sick, they'd look into more of that.
However, reproducibility is absolutely vital. If future researchers can't replicate the results of a study, then statistical flukes are going to be a real problem. If studies can be replicated, then statistical error can be handled very efficiently. Each time a given piece of research is reproduced (even if it's only reproduced to serve as a set-up control for a future study), will exponentially reduce the odds of a statistical fluke surviving.
Does anyone have this reference? I can't find it again.
Gravity is proved. Evolution is rock solid. We still don't know if coffee is good or bad for us.
That's not the hard part. In the humanities, people care about the answers. They want to find economic value in diversity, for example.
No one went into medicine to prove that a particular compound cured gout.
In the hard sciences, there is rarely an agenda. People often care very much about the results, but they rarely bring with them deeply rooted political biases.
Gravity is not proven. Falling is easily observed, and general relativity is a pretty good theory for it, but the jury is still out on how exactly gravity works. Evolution is also easily observed (e.g., drug resistant pathogens); its basis in natural selection and genetics is the part that is rock solid.
As for coffee, it's some combination of good and bad. Which part dominates in which situations is up for debate.
(Apologies for being pedantic -- caffeine high just set in.)
Sure, thats old physics/biology. You can't compare that to state of the art medicine. You need to look into modern cosmology, string theory, quantum mechanics and make the comparison. Unfortunately, I'm not in a position to make such a comparison.
That aside, the incentives are different. Medicine --> $$$. String theory doesn't.