Simpson's Paradox
vudlab.com
vudlab.com
The almost universally omitted variable is health-consciousness. Some people are health-conscious and some aren't. People who are health-conscious do a whole bunch of things, some of which help (like exercise, sleep well, eat moderately). They also do things that are widely believed to be good for you, like eating broccoli.
So if you do a study, you'll find that people who eat lots of broccoli are healthier. You'll be able to confirm pretty much any widely believed health folk wisdom, unless it's something quite harmful, as long as you omit health-consciousness as a variable.
A simple example would be a wet lawn. We know rain causes a wet lawn, and our observation shows indeed that rain and wet lawns are strongly associated. However, observing a case where a given lawn is wet, and yet there's no associated rain is a clear signal that a latent cause hasn't been accounted for (namely sprinklers). This principle still applies in noisy observational data or probabilistic rather than deterministic causal relations, though you do need a bigger sample to reach the same confidence.
We also can look at measures of model fitness. If a variation of the model that hypothesizes a latent causal variable is more likely to generate the observed data than the model without it, we know we've missed something. The general case of this is that we learn the model itself from the observed data.
This page from Kevin Murphy is a reasonable survey of the methods: http://www.cs.ubc.ca/~murphyk/Bayes/bayes.html I also recommend his textbook.
Epidemiology studies can be randomized or observational.
That is, if they can control for it, they can make assertions about smaller effects.
Why would you want to do that? You can build a nice career on publishing (spurious) associations. As long as relevant variables are omitted, no epidemiologist need ever be unemployed.
If your colleagues can easily see that your statistical analysis is flawed, they won't waste their own time doing work that builds off of yours.
See e.g. Chan and Boliver "The Grandparents Effect in Social Mobility", http://asr.sagepub.com/content/78/4/662 (HT http://www.arnoldkling.com/blog/a-grandparent-effect/) and many of the works that it cites. There seems to be a sizable mutually-citing "scientific" literature devoted to studying correlation in biological descent while dogmatically not controlling for ordinary DNA-based inheritance of psychological traits from parent to child. (Indeed, not even controlling for DNA-based inheritance of physical traits. Health and height and attractiveness are correlated with income, and are significantly physically heritable.) By not controlling for the properties which are transmitted by sperm meeting egg, your can find all sorts of fascinating possibilities to investigate to explain the correlation. (E.g. Chan and Boliver carefully note that "well-connected grandparents could also use their social contacts to help grandchildren with job searches." Good, good, carefully investigate all the possibilities.)
Nope! This is how we might like to to work, but cool associations will get written up forever, long after they have been refuted by randomized experiments. (Heck, papers can outright be retracted and still get cites.)
Hahaha - did I say refuted by randomized experiments? Like that ever happens to more than a tiny fraction of epidemiology correlations... No, one's career is quite safe. No one ever lost tenure because the correlations they built their career on turned out to be lame.
> If your colleagues can easily see that your statistical analysis is flawed, they won't waste their own time doing work that builds off of yours.
No, the point here is that your work can be immaculate and still never reflect causality. How are you going to collect data on every lurking variable? You can't, of course.
It occurred to me a while after my other comment that a belief in the ability of people that test as health conscious to actually do things that improve their health might not be especially justified. I mean, lots of people are quite happy with whatever quackery, maybe it really does all come out in the wash.
It was an eye opener for me.
(I realize that doesn't make any guarantees about common practices, but it is maybe a little capricious to repudiate the entire field)
> Every ommitted variable problem
Not quite the same thing as the OP (not sure if you were implying it was). Regardless, OVP is still a great thing to have in your mental framework.
In short, Simpson's paradox occurs because probability distributions and causal claims are distinct things which behave differently. It's nothing more than a particular, insidious example of correlation not implying causation. It's perfectly possible for a probability distribution to have a "contradictory" shape, but perfectly impossible for logical statements about the world to be contradictory.
The resolution is that you shouldn't let your probability distributions turn into logical statements without analyzing your causal assumptions. This will lead you to whether or not excluding a variable is omitted variable bias (and whether including an improper one will lead to included variable bias, which is rarely recognized).
What is the meaning of the green and purple lines on the graph, why do they have different gradients and why can't I adjust them? Why does the Simpson's Paradox apply sometimes and not others? Why are there so many bars and donut charts? What does the gray circle around the donuts mean? Information overload.
In the doughnut charts, blue is accepted rate, red is rejected rate (as labelled earlier in the article).
The gray circle indicates which has a higher percentage of acceptance between male and female.
The y-axis on the line graph is %age admitted combined over both departments. As you drag the women slider, you'll see the %age accepted rate for women combined over both department matches the y value of the women line in the line chart.
The illustration is to show how the distribution of applicants between the easy and hard departments (the lurking explanatory) effects the combined acceptance percentages (the explained). This is the focal point, so all other variables need to be fixed.
The fixed values are: in the easy department, 62% of men are accepted and 80% of women are accepted. In the hard department, 26% of men are accepted and 27% of women are accepted. In both departments women are slightly favoured over men, so both always have a gray circle around the women's charts.
There are 1,835 women applicants and 1,362 men applicants, making a total of 2,691.
The combined accept rates is derived from this static data and the user controlled variable of the distribution of applicants between the departments. You can pick some arbitrary inputs and trace the maths through to calculate how many apply to each department, and how many of those get accepted, then add that up to find the combined value (be warned the actual percentages aren't the nice rounded ones they label them as).
While the point on the purple line is below the point on the green line in the y axis, the combined acceptance percentage for women is larger than the men. There is no Simpson's paradox here because taking the lurking explanation into account - that the distribution of applicants between departments has an effect - does not change the outcome. Women are favoured in all three measurements; the gray circle is around all the women's charts.
When the point on the purple line is above the point in the green line in the y axis, the combined acceptance percentage for women flips to being smaller than men. Here we have the Simpson's paradox - taking into account the lurking explanation changes the outcome. Women are favoured in the breakdown by department, but men appear favoured in the combined statistic.
I found the interactive chart helped me figure out this explanation, but it wasn't helped by poor labelling and the fact they seem to have used real (read messy) statistics to perform the underlying calculations with rather than the nice rounded "62%" etc. they label.
Compare and contrast to the clarity of their central limit theorem demonstation [1] vs its wikipedia page [2].
The donuts indicate which group (men or women) was admitted more in each kind of department.
Play around with the sliders and you can see it toggle in the combined row, while simultaneously a yes/no answer updates after the question "Simpson's paradox?" It would really help if yes/no was highlighted. Basically, you can make the paradox appear by dropping the % of women applying to easy departments. You can't make the donut switch in easy or hard departments individually... that's the whole point: women have a slight advantage in each.
Honestly, I found the "Proper Pooling" section to have the most intuitive explanation. Combined, there is a bias. By department, there is no bias. The reason is clear: very few women applying to easier departments.
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.34....
An oversimplification of his idea would be to say that given assumptions about the causal independence of variables, it becomes clear which way you should group the data. Although it's not always possible to make these independence assumptions, much of the time they are obvious and uncontroversial.
Taking the Berkeley gender bias case as an example: We know it's possible that biological gender influences which department graduates apply to, but that it's impossible that the department a graduate applies to influences their biological gender. This fact alone tells us that we need to look at the data by department rather than in aggregate, resolving the paradox.
But this is a great, fast example of why his work is important.
[0]: http://www.villainouscompany.com/vcblog/archives/2012/04/chi...
Then that article places the claim "Additional evidence, however, indicates that women may be less able to afford the lawyers and experts needed in contested custody cases (see “Family Law Overview”) and that, in contested cases, different and stricter standards are applied to mothers." without providing any data. So it ends up being a propaganda piece, which makes it not very trustworthy.
Lecture 6 talks about this paradox and if I recall correctly he might even talk about this exact case:
https://itunes.apple.com/us/course/statistics-110-probabilit...
But I think this last statement
"or Texas schools to waste money copying Wisconsin."
was meant to be the other way around
Overall test scores are higher in Wisconsin only because white students in both states are the highest scoring group, and Wisconsin schools have a higher proportion of white students than Texas schools.
To the extend that any of this can be copied, it would probably be better for Wisconsin to improve the test scores of its ethnic groups to Texan levels, rather than for Texas to emulate Wisconsin's lilly-whiteness.