Comment by nkurz | Hacker News Reader

Comment by nkurz | Hacker News Reader

I worked on a project a few years ago for a UC Berkeley biostatistics professor who believes he has a theoretically proven approach of using non-randomized observational data to create an asymptotically-efficient unbiased estimator of counterfactual treatments. My impression as a programmer who has only dabbled in machine learning is that this is a phenomenal claim far beyond the state of the art.

My job was to take an inefficient proof-of-concept R package, and make it computationally and memory efficient enough to run on real-world datasets. I failed totally. The obvious explanation would be that I just wasn't able to understand the math involved well enough to implement the algorithm despite immersing myself in it for months. My personal guess though is that both the paper and the reference implementation were flawed in some way that made the task impossible.

Anyway, the paper is here:

Targeted Maximum Likelihood Estimation for Dynamic and Static Longitudinal Marginal Structural Working Models

Petersen, Schwab, Gruber, Blaser, Schomacher, and van der Laan; J Causal Inference 2015

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4405134/pdf/nih...

And the R package here: https://github.com/joshuaschwab/ltmle.

My understanding was never perfect, but my belief is that there is kernel of insight in this approach that has not yet been explored in machine learning. Alternatively, maybe it works as is, and just needs a better implementation. I'd love to see someone implement this approach, or fix it, or discredit it. As it is, I think it's potentially incredibly valuable work that is getting very little attention.

> [...] who believes he has a theoretically proven approach of using non-randomized observational data to create an asymptotically-efficient unbiased estimator of counterfactual treatments

That would be exceptionally powerful if applied to the social sciences. I've long thought that the extreme lack of statistical rigor in that field has led to all kinds of damaging and biased social policies that negatively affect society, ranging from the legal system to the criminal justice system.

This is very interesting from a development perspective. You seem to have tested the working parts of the algorithm and broke it down to multiple passes when appropriate, I am surprised your work did not reach completion. You did all the right things.

What exactly was considered wrong with the output? How did this fail?

How did this fail?

I'm still trying to understand this, and I worry that much of it is due to my personal failings. I think I'm much worse than most people at making headway on problems when I know that my understanding is flawed, and I rationalize this as part of my moral code as a sort of "first do no harm". But I'm scared to discard my moral instincts for sake of convenience, out of fear that I wouldn't know where to stop.

In this case, I couldn't find an interpretation of the paper that matched my interpretation of the sample code, and I couldn't find an interpretation of either that seemed plausibly correct. The flaws in each seemed so obvious to me that I had to presume that I was not interpreting either correctly, and I concluded that I must be missing something essential.

There wasn't much in the way of test structure other than eyeing up the results and declaring it to be OK. I was scared that if I inadvertently implemented something incompatible with both the paper and the code, my errors would never be caught. None of the grad students seemed to fully understand the paper, and the professor wasn't familiar with the R code. My clumsiness with the terminology of the field made it difficult for me to communicate with the professor.

I feel terrible about the whole thing, but don't know what in particular I should have done differently.

Hey, the key assumption (the "identification" assumption) is in 2.4 of the paper you referred to. I do not have time to go through the notation, but it seems to be like a fairly standard "no unmeasured confounders" assumption - so you need to ASSUME that nothing influences both the treatment and the outcome you are interested in. Then, of course, you do not need to randomize, but can use observational data.

I know that Petersen and van der Laan are well-respected researchers in causality that know exactly what they are doing. E.g., Petersen has a course (I think at Berkeley) that uses causal graphs which were pioneered by Judea Pearl (see comments below). I can only second the recommendation to dig into his work.

can you explain to me why you think this result implies "asymptotically-efficient unbiased estimator of counterfactual treatments" ?

Reply on news.ycombinator.com