Medicine’s Uncomfortable Relationship with Math (2014)
archinte.jamanetwork.com
archinte.jamanetwork.com
You can read an example (http://science.sciencemag.org/content/351/6277/1037.2.long, http://www.be-md.ncbi.nlm.nih.gov/pubmed/26941311/#comments) I linked in another comment (https://news.ycombinator.com/item?id=11325254). There are quite a few blogpost on this also, and if you allow me to shorten the story in one sentence, that is a comment on Science arguing a previous study on Science did the statistical analysis wrong is under the critics that the comment itself did statistical analysis wrong.
And recently life sciences fields are in a "reproducibility" controversy which basically arises from education and application of the topic.
The first time I got a biostats lecture in medical school, I genuinely questioned their advice. By the third class, I was convinced I knew nothing about math. Luckily our biochem prof was a physics major and assured me I didn't need to worry about the stats.
Somewhere in residency, I realized I really had been getting all the stats questions right. But not because I did anything like the other residents. They have all these crazy stories and mnemonics to keep track of what kind of problem they're looking at. They seriously try to classify their way out of basic math. They classify. Everything.
I now agree with my profs. I didn't need a stats class. But the stats education in medical school seems designed to convince physicians that they need statisticians. Which is probably a coup for biostatisticians, but it's a damn shame for physicians trying to break into research outside major academic centers.
Relevant to your main point - I have worked with many clinicians. Most of them have had a fairly tenuous grasp of the statistics they were working with, which matches your experience in training I think. A couple had much better understanding, but had driven that education themselves.
It is true that depending on how statistical mechanics is taught, it can be difficult to connect to other statistical reasoning (the distributions are "nice", you have a gazillion measurements, and your population sampling doesn't tend to have bias issues, etc.)
How can you possibly say that? Are you familiar with the course prerequisites for every biology and medicine degree in the world off the top of your head?
And quite frankly, the PPV trope is just that, a trope. The biostatisticians found a little trick to lord over their masters. If you only have 15 seconds to solve after reading the question, and you know you only do this once every 6 months, and you know most people get it wrong, answer C, next question.
Christ, the story that docs suck at stats is part of medical education. They don't teach biostatistics, they teach "you suck at biostats".
I've had to do a lot of extracurricular statistics to get up to speed but a disturbingly large portion of people don't. People generally know t-tests and maybe ANOVA, and that's about it. Of course they don't know any of the theory behind it and often just apply t-tests to everything, even when inappropriate.
In this case we write down the contingency table. Assuming that the test perfectly detects what we are looking for we find
True positives: 1
False positive: 5% of 1000 = 50
True negative: 949
False negative: 0
Chance of disease given positive results = 1/51 = 1.96%
But as you said, under the assumption that the test perfectly detects, the results are correct.
- Out of 1000 tests 51 will be positive, 50 of which are incorrect.
- Out of 1000 positive tests, 50 will be wrong, 950 will be correct.
The 2 interpretations give vastly different results.
"Out of 1000 positive tests, 50 were false positives, 950 were true positives" - valid statement.
"The false positive rate was 50 out of 1000" - abuses a common technical term in a way that sounds valid on the face of it, but which is potentially VERY misleading.
We can't even calculate a valid false positive rate from the above data, since that requires taking the ratio vs. all tests and not just positive tests.
I don't see anything here that definitively clarifies which of the 2 scenarios above it can exclusively be applied to.
Dividing by the number of positive tests instead gives what's called the 'false discovery rate' which is pretty rarely used.
FP / (FP + TN)
Where FP is number of false positives, and TN is number of true negatives. So it's a third option:
- Out of 1000 actually negative samples, 50 were tested as positive.
So in the case of 1000 samples, 949 correctly testing as negative, 50 incorrectly testing as positive, and 1 correctly testing as positive, the false positive rate is 50 / 999.
False Pos. Rate = P(predict + | state -)
= P(predict + and state -) / P(state -)
= P(predict + and state -) / (P(predict + and state-) + P(predict - and state -))
~ #FP/samplesize / (#FP/samplesize + #TN/samplesize)
= #FP / (#FP + #TN)
So to detect false positives in a mathematical way, you should apply the test only to sane people and now the proportion positive/total is an estimation of the false positive rate.
Cited by 272(!!): https://scholar.google.com/scholar?cites=1812909520721081729...
What is a bit sad is that data analysis and quantitative skills continue to be de-emphasized in the pre-clinical curriculum for the sake of rote memorization of huge lists and tables, which would seem to be less and less important as instant access to references on smartphones becomes commonplace.
Where it breaks down is medical science, which relies so much more on subtle mathematical interpretation than most other empirical sciences. But that is just what happens when there is so little opportunity for deliberate experimentation and such a large dataset of "naturally occurring" experiments to observe. (and it surely is helped by this millenia old tradition of nonempirical medicine, i guess all that bloodletting based on beautiful but hilariously wrong theories is to this day giving the theory-building part of the scientific method a bad reputation in medicine, making it work completely in "stupid mode empirism" where every part of thinking that is not strictly looking at numbers is heavily frowned upon)
Maybe there is a need for a stronger differentiation between academic paths leading to to applied medicine and those leading medical science. Biotech comes close, but it certainly isn't about teasing more reliable insights from clinical datasets. Other sciences take the easy road and simply train everybody for science, leaving practical application to learning on the job, but we sure would not want to get treated by that kind of MD.
When I was a pre-med student, for some reason the prerequisites required a year of calculus. That succeeded in weeding out people who can't make an A in freshman calculus, but I'm not sure what else it accomplished. Calculus has little to do with the daily practice of medicine, unless you're a radiation oncologist or doing some hardcore research.
A year of statistics would have served me and my patients much better. That goes double, given the current firehose of data that is part and parcel of the personalized medicine revolution.
My experience has been that big data is more about linear algebra, which is usually several classes beyond entry level calc or stats. You have to be able to reason about arbitrarily large collections of partial differential equations (albeit reduced to difference equations).
For example, if you want to talk about genomics, Durbin's Biologic Sequence Analysis is probably the most foundational text available. It introduces Bayes's theorem on page 8, has stuff that looks suspiciously like calculus on page 40, and is into Markov chains by page 48. They hold off on a formal treatment of entropy until about half-way through the book.
And for phylogenetics, the equivalent books is Felsenstein's Inferring Phylogenies. He introduces linear algebra before integration.
My favorite quote from Felsenstein, particularly germaine for pathologists (surely the taxonomists of medicine):
"Knowing exactly how many tree topologies ... is not particularly important. The point is that there are very large numbers of them. ... one use for the numbers was "to frighten taxonomists."
I've been trying to recognize and account for this deficiency by drawing mental decision trees enumerating probabilities instead ... anyone else doing anything similar?
However, in probability thing can quickly get complicated and, for me at least, intuition can't always be trusted. (I probably needed to go to the lectures and read the book.) For examples see either Schwarz's 40 Puzzles and Problems in Probability and Mathematical Statistics or Grimmett and Stirzaker's larger One Thousand Exercises in Probability.
And those look like excellent exercises as well; thank you doubly :)
If you start w/ 50 red candies and 50 non-red candies, the chance of the first one red is 1/2 (from 50/100) but the chance the second one is red after first drawing a red candy is now 49/99. After removing one red candy, there are now 49 red candies and 99 total candies.
So, P(red) * P(red | removed 1 red) = 49/198