Hacker News Reader

Comment by TheMrZZ | Hacker News Reader

Biggest trap of Simpson's paradox is the results can change with every level of granularity.

If you take the example of Treatment A vs Treatment B for tumors, you can get infinite layers of seemingly contradicting statemens: - Overall, Treatment A has better average results - But if you add tumor size, Treatment B is always better - But if you add gender to size, Treatment B is always better - But if you add age category to gender and size, Treatment A is always better - etc...

It totally contradicts our instincts, and shows statistics can be profoundly misleading (intentionally or not).

View on HN

To add some proofs to my answer, I actually coded a Z3 program to prove it! The 3-variables version takes too long to resolve, but I got results for the 2-variables version (tumor size + gender):

Results can be found in this GSheet: https://docs.google.com/spreadsheets/d/1tsBhElTgXjVTeas8quar...

Code is here: https://gist.github.com/TheMrZZ/c33927ca2cc917997a67d7f84b82...

I'm currently running the 3-variables version, hopefully I'll get results this afternoon.

We can clearly see the same problems that arise in the 1-variable Simpson's paradox (widely different population sizes).

Like Tumors are higher dimensional objects than what human brains are trained to perceive.

If the previous comment is right, then this one is plainly also true in some sense. I'm disappointed to see downvotes.

> If the previous comment is right

I actually coded a Z3 program to prove it! The 3-variables version takes too long to resolve, but I got results for the 2-variables version (tumor size + gender):

Results can be found in this GSheet: https://docs.google.com/spreadsheets/d/1tsBhElTgXjVTeas8quar...

Code is here: https://gist.github.com/TheMrZZ/c33927ca2cc917997a67d7f84b82...

I'm currently running the 3-variables version, hopefully I'll get results this afternoon.

We can clearly see the same problems that arise in the 1-variable Simpson's paradox (widely different population sizes).

For pedagogues and practitioners alike: there is a subtle connection between Simpson’s paradox and the wild geometry of relative entropy. This might be partly why effect sizes are also contentious.

Besides Ellenberg’s mind-altering discussion of that link[1], see hints on the second page of:

https://www.qeios.com/read/XB1N2A/pdf

[1] "[the point of Simpson’s paradox] isn't really to tell us which viewpoint to take but to insist that we keep both the parts and the whole in mind at once."

Ellenberg, from Shape: The Hidden Geometry of Information, Biology, Strategy, Democracy, and Everything Else (2021)

I think the real-world resolution to this problem is straightforward though. You should look at the finest level of granularity available, and pick the best treatment in the relevant subpopulation for the patient.

Unfortunately our level of certainty generally falls off as we increase the granularity. For example, imagine the patient is a 77yo Polish-American man, and we're lucky enough to have one historical result for 77yo Polish-American men. That man got treatment A and did better than expected. But say if we go out to 70-79y white men we have 1,000 people, of which 500 got treatment A and generally did significantly worse than the 500 who got treatment B. While the more granular category gives us a little information, the sample size is so small that we would be foolish to discard the less granular information.

This is all true. I originally added a disclaimer to my post that said "assuming you have enough data to support the level of granularity" but I removed it for brevity because I thought it was implied -- small sample size isn't part of Simpson's paradox. My apologies for being unclear

The smaller the subpopulation, the higher the variance, and the less significant the result.

Reply on news.ycombinator.com