Simpson's paradox

The most fascinating aspect is that once you recognize the issue, you can't simply rely on using just the partitioned data or the aggregate. Be sure to read the "Implications to decision making" section.

http://en.wikipedia.org/wiki/Simpsons_paradox#Implications_t...

The practical significance of Simpson's paradox surfaces in decision making situations where it poses the following dilemma: Which data should we consult in choosing an action, the aggregated or the partitioned? In the Kidney Stone example above, it is clear that if one is diagnosed with "Small Stones" or "Large Stones" the data for the respective subpopulation should be consulted and Treatment A would be preferred to Treatment B. But what if a patient is not diagnosed, and the size of the stone is not known; would it be appropriate to consult the aggregated data and administer Treatment B? This would stand contrary to common sense; a treatment that is preferred both under one condition and under its negation should also be preferred when the condition is unknown.

On the other hand, if the partitioned data is to be preferred a priori, what prevents one from partitioning the data into arbitrary sub-categories (say based on eye color or post-treatment pain) artificially constructed to yield wrong choices of treatments? Pearl[2] shows that, indeed, in many cases it is the aggregated, not the partitioned data that gives the correct choice of action. Worse yet, given the same table, one should sometimes follow the partitioned and sometimes the aggregated data, depending on the story behind the data; with each story dictating its own choice. Pearl[2] considers this to be the real paradox behind Simpson's reversal.

As to why and how a story, not data, should dictate choices, the answer is that it is the story which encodes the causal relationships among the variables. Once we extract these relationships and represent them in a graph called a causal Bayesian network we can test algorithmically whether a given partition, representing confounding variables, gives the correct answer. The test, called "back-door," requires that we check whether the nodes corresponding to the confounding variables intercept certain paths in the graph. This reduces Simpson's Paradox to an exercise in graph theory.

I think this and "Anscombe's Quartet" http://en.wikipedia.org/wiki/Anscombe%27s_quartet are great examples of why you should look at your data before trying to analyze it.

The "Berekley gender bias case" section made it make sense to me. Every single department was more likely to admit a woman than a man, and yet the school as a whole was more likely to admit a man than a woman.

This was because women were applying to more competitive departments, on average, so a lower percentage of women applicants were getting admitted.

I like how the article uses "Lisa" and "Bart" for people’s name in the example: http://en.wikipedia.org/wiki/Simpsons_paradox#Description

I think the take-home here is summarized here:

This imagined paradox is caused when the percentage is provided but not the ratio. In this example, if only the 90% in the first week for Bart was provided but not the ratio (9:10), it would distort the information causing the imagined paradox. Even though Bart's percentage is higher for the first and second week, when two weeks of articles is combined, overall Lisa had improved a greater proportion, 55% of the 110 total articles. Lisa's proportional total of articles improved exceeds Bart's total.

So it's important to not rely on percentage data alone, as it is a form of reduction. Rely instead on ratios, allowing you to see the validity of the percentage.

On Amazon and review sites, when I search by rating, I always look at the number of reviews and weight accordingly. I remember that IMDB (before it became an Amazon property) implemented a beyesian posterior mean to cull out the low-review anomolies: https://secure.wikimedia.org/wikipedia/en/wiki/Internet_Movi...

I wish other places like Amazon would implement similar weighting mechanisms to really allow a user to navigate by reviewed quality.

Unintended consequence of reading the article: I need to rethink decades of conditioning. I thought Republicans opposed the 1964 Civil Rights Act.

Simpson's paradox is amazing, it's worth reading all the examples in the article and really understanding it.

Another good source is Martin Gardner's "Aha! Gotcha" book. He presents the paradox as a woman trying to find eligible bachelors at a party. In room 1 her odds are better if she goes for guys with mustaches. In room 2 her odds are also better if she goes for guys with mustaches. But when everyone goes into one room, her odds are better for guys without mustaches. It's incredible.

It's not really that interesting, and the paradoxical/unexpected aspects of it go away once you start looking at everything through the lenses of conditional probability theory. e.g. http://uncertainty.stat.cmu.edu/ goes over it as early as chapter 2.

This happens to students on a fairly consistent basis.

Suppose a project is worth 10% of the student's final mark while a midterm is worth 30% and the final exam is worth 60%. If the student performs well on the project, average on the midterm, but poorly on the Final Exam, their mark is still going to be poor due to the heavier weighting of the exam and midterm over the project.

EDIT: I know it's not a perfect example, but comparing marks between different students based on how they are weighted is essentially the idea.

"A real-life example is the passage of the Civil Rights Act of 1964. Overall, a higher ratio of Republicans voted in favor of the Act than Democrats. However, when the congressional delegations from the northern and southern States are considered separately, a higher ratio of Democrats voted in favor of the act in both regions. This arose because regional affiliation is a very strong indicator of how a congressman or senator voted, whereas party affiliation is a weak indicator."

The chart then shows that the "Northern" House had 316 members vs. the South's 104, and that the "Northern" Senate had 78 members vs. the South's 22.

Uh... the South has Florida, Texas, and California. How can it be represented by less than a quarter of Congress? I mean, the Senate makes a bit more sense due to the tiny states of the northeast and such (though it still seems low), but the House too? Really?

Makes me wonder how exactly they classified South vs. North... Former Confederate states vs. everyone else? Either way, Geography cannot be a major factor.