Causal Inference Book

Judea Pearl also wrote a summary of The Book of Why here: http://ftp.cs.ucla.edu/pub/stat_ser/r481.pdf

There's also the book "Why, A Guide to Finding and Using Causes" by Samantha Kleinberg http://www.skleinberg.org/why/

My theory: It’s a touchy subject. It has not been inhibitory up to this point (mid late 2010s) to divide things into things that follow laws, and things that are complex intersections of many systems of social power and where each individuals personal epistemology must be considered in the context of a holistic analysis that is not loaded with preconceptions about values we acquired during socialization and now that you can process fifty exabytes of data did you even think about what if your programs aren’t accurate representations of reality, sheesh this is why engineers shouldn’t make public policy

Case in point: other comment links to Pearl’s book. He has been writing about general causality for decades, but his latest book shows he realized there is a desperate need to communicate the basics to people who like the modes of thinking that cause them to avoid trusting mathematical “solutions”

Typically, stats seems to have different usage and teaching by field. Psychologists have their flavor, physicists another. Econometricians, epidemiologists, nutritionists, etc, all different. They're pretty much all asking causal questions, but they're coming up with different techniques to solve the specific weird questions that are unique to their field.

Some fields get to randomize everything. Causal inference is is as simple as asking "is this difference real" when the difference in your experiment would be causal by design. This is what is taught as stats 1 to everybody who starts learning stats.

Many fields don't get to randomly do things :(. Maybe you want to know the effect of a disease, but we as a society have decided that harming people to learn for the eventual greater good is bad (hello "rush out the self driving cars!" proponents!). Or maybe randomizing would involve merging a few multi-billion dollar companies and grad students don't have that budget. For whatever reason, you can't intervene in the same way. I give those two examples because epidemiologists and economists have really extended the state of the art of causal inference because their data doesn't look the same as many other fields. Inference is harder (but still possible!).

Those niche developments have been happening for a while, but haven't been cohesively un-niched. Big (non-randomized) data is becoming a big thing and everyone wants to use it to make decisions. Decisions require causal statements and most people only know how to make those statements for cleanly randomized stats 1 stuff. People seem to be picking up on the fact that some of this previously niche stuff is much more broadly useful.

It may be wrong in the sense that free will is wrong, but a lack of free will doesn't stop me from making decisions. Causality is all about counterfactuals, and the admission is right there in the name counter-factual. We are already aware.

The most popular framework for causality talks about potential outcomes. If I took my blood pressure meds now (all else equal), what would my blood pressure be at noon? If I skipped my meds today (all else equal), what would my blood pressure be at noon? That tells me the casual effect of the meds on my blood pressure today. But I either will or won't take the meds. One of those branches is simply wrong on a fundamental level, like you say. "What would happen" is in some sense meaningless. But it's sure as shit useful to think about, and that's the whole point.

Said differently, even in a toy model of the universe where everything is deterministic and causality is meaningless on a physics level, causality can be usefully defined on a practical level. Counterfactuals are counter factual, but that's ok.

Good luck predicting complex, real-world systems by their subatomic constituents.