Comment by mjburgess | Hacker News Reader

Comment by mjburgess | Hacker News Reader

The issue here is the same across all 'science' which is unable to effectively control causes, and which studies non-linear phenomena (or, more simply, cases where the distribution of means isnt defined, or is far away from normal).

The issue is: you can't do stats. The reality is it is highly likely that all data here is consistent with any effect size you care to choose.

What this study does, nevertheless, is quantify the 'model risk' ie., the "p-value of the method itself" -- how often does research provide reliable effect sizes? Rarely. So the research methods themselves have a huge 'epistemic discounting' associated with them.

But we can go much further than this -- recurse: what is the risk of using this methodology to assess methodology? Actually, it's fairly high. This isn't a very reliable method for judging research methods. And so on.

What we find here is that in the vast majority of cases, where we'd care to research anything, recursing risks lead to an essential insurmountable error: measurement error, model error, methodology error, research-assumptions-error, etc. There isnt enough data to resolve between competing possibilities at the outer-most stage of error (often you'd need much more data than practically possible to collect).

What do we do with this? I'd say, mostly, throw away stats here. Do case analyses, risk-based recommendations, even broadly philosophical analysis.

The alterantive is just staring at tea-leaves.

You're not wrong, but I think you may have headed out the wrong way from your conclusion.

Reproducibility trumps causal models. If you call the psychic hotline and get the same results time-after-time, then keep using the psychic hotline. Leave the modeling to others.

When developing new science, stats lag behind reproducibility. First you see something that correlates, then somebody is able to make A always infer B, then you start doing some statistical surveying in an effort to build up a larger model.

Observation first, then Abduction (in the Peircian sense), then measurement, then modeling, and finally Deductive/Inductive reasoning.

We keep trying to do it backwards where we have the model for everything, then bang actual data into various parts of the model. The saddest thing in this process is that if we're only doing incremental, normal science (Kuhn), none's the wiser. This is why reproducibility tests are so important. We lie to ourselves. We're quite good at it.

BTW, I agree that with you general conclusion about case analysis and such, but only in the sense that it drives Abduction, not as a goal in itself.

The case analysis would drive, what i'd hope to be, "humble abduction" rather than the variety we've become acustomed to.

Ie., rather than say "vaccines cause autism" we'd say: here are some cases where they co-occur; here is an empirical investigation into possible causes in these cases; here's the risk profile across various possible 'abductions'.

Were we to lack any good science of immunology, etc. we'd have to stop there and then start a risk analysis assuming each possible universal conclusion was true, hence giving a kind of "policy ensemble".

Since we have good-enough immunological science, and so on, we can show that the lack of anything other than time-correlated instances makes the likelihood of a causal relationship near zero (in epistemic risk terms, at least).

In otherwords, I expect for most areas we're interested in, there is never going to be anything close even to immunology -- let alone physics. I dont think many of the world's universals are statable.

Namely, i think chaotic non-linear systems preclude it; and systems with power-law like behaviour make collecting enough data practically impossible.

I'm sceptical there will every be anything more than sociology for society; literature for articulating inner psychological experiences; etc.

This is, in part, why stats is doubly harmful in these cases. It isnt a kind of protoscience, it's acutally pseudoscience which gets in the way of 'humble abductions'

I've been using a phrase for several years, "profound ignorance". I might use "humble abductions." Thanks.

We always end up back with cluster/cohort analysis, then definitions. I think a big part of the problem is that we've already convinced ourselves of the definitions of the terms. These definitions prevent actual progress. We end up defining ourselves into our own locked room.

I think the first task in this sort of convo is to convince the audience (client) that we are necessarily profoundly ignorant. You cannot predict prices, you cannot predict interest rates, and so on --- its still always a little suprising to see the shock in their eyes.

No, indeed, the thing you've been asked to do is an exercise in risk management -- there is no statistical reality to the predictions here; ignore them.

So, i'd still say that opener is vital. I try to move on to ensemble scenario modelling prose-style by several experts, etc. then, eventually, i tell them about geometric brownian motion -- only when i'm convinced they're absolutey certain it's smoke-and-mirrors

And that therefore they are aware that their job is primarily to manage risk, and a statistical model is only one minor (often blinding) tool in that box

If you need a point estimate, so be it, but claw it from my cold dead fingers. Most of the world isnt knowable, sorry to disappoint. Now go actually manage your damn responsibilities

To continue mining the Peircian vein, it appears to me that the most troubled areas of inquiry are the ones where the pragmatic maxim gives us the least.

Or compensate by expecting much, much larger sample sizes than are currently typical?

Reply on news.ycombinator.com