None of the Above | Hacker News Reader

None of the Above

elevanth.org

37 pointsTomte18 comments

In grad school, I had conversations with a couple different conversations with fellow PhD students in the life sciences. The question I asked was not about p-values / hypothesis testing but about the (lack of use) of something else technical (I don’t recall the specific question). They all responded, “we haven’t done math since our intro to calculus class and we don’t know how to code.”

I feel like that is really at the heart of the problem. It is difficult to create the kind of bespoke scientific model required and use it to perform statistical analysis when you lack the mathematical and computational background. If you don’t know what you’re doing, it’s very easy to misapply existing statistical tests, let alone understand why you might care about something like the bias and variance of your estimator. It’s an understandable problem — if you are a biologist, you want to think about biology, not statistics and most biology work doesn’t require math so it appeals to the scientifically-minded that dislike math and programming.

I’m curious if people here who studied those subjects think this intuition tracks, or if I am way off base. If it is part of the problem, I don’t know what the solution is beyond requiring more mathematical and computational literacy, which likely comes at the cost of discipline-specific material.

I have a background in economics and finance with a stronger-than-average focus in statistics (compared to my peers in these fields), and I agree with you that statistical literacy (both in terms of application/interpretation but also an understanding of the mathematical foundations) are lacking in not only life sciences but really any field besides statistics. I am certain I must have done some bad-stats with the research I worked on too, because as the article also points out, it's hard to do it right and be certain you have a correct model with appropriate tests (even with so many eyes on it!)

Increasing the literacy requirements is an option, but not realistically implementable. I think providing guidance and implementation support by actual practicing statisticians being involved in the data analysis sections of published research as a prerequisite is more realistic.

A process in which experiments are conducted and data is collected, and the statisticians are involved in the interpretation in collaboration with the researchers is expensive in time and money, but I guess will produce much better quality analyses.

Who wants to put this one to test?

I agree that collaborating with dedicated statisticians is more feasible. I think part of the challenge might be ingraining in researchers that “stats is both hard and important so you need to take it seriously or find / hire someone who does.” If they don’t appreciate the problem, they won’t seek a better solution.

Have been there. They blame the statistician getting in their way the paper published.

So I'm an epidemiologist and/or a disease ecologist depending on what hat I'm currently wearing, so I think I count as someone who can address your question.

And I'm going to give you the epidemiologists favorite answer: "It depends."

Having a math and computation background certainly helps. There's a conference I go to every year that I refer to as a hypothesis generating conference - I go and listen to talks, and write down ways to address problems primarily with computational models that the presenters (mostly clinicians) think are intractable.

But I've also seen people with very strong math and computation backgrounds run completely aground, either by being in love with a method more than a question, or just utterly lacking subject matter expertise.

I think there's two more cultural things that are plaguing science, that don't necessarily get fixed by "take more calculus":

1) A desire for there to be rote guidelines. "In case of X, I do Y." A lot of the push recently (see the "Use confidence intervals, not p-values" or preregistration) are about making better rote guidelines, but they're still rote. As an example, the JAMA journals prohibit the use of causal language unless you're reporting an RCT.

Which is mostly fine. Except, on occasion, when you get a mathematical model accepted into a JAMA journal. A model that is inherently causal, but doesn't fit "the list", and you end up writing the equivalent of "Two plus two is associated with an outcome of four."

2) Publishing. But not in the normal way people bring up publishing. There's a desire for papers to be a complete story. To be done. The experimental sciences sort of leaked into everything, so there's a notion that all studies have a beginning, middle, and end that closes the book on whatever small bit of a project you're working on, versus something closer to living progress on a question.

Thanks! Yeah I should clarify that these friends I were talking to do the kind of bio / neuroscience where they spend most of their time in a lab running a physical experiment so their skills are focused around that kind of wet work and they interact very little with mathematical folks based on what they implied.

Also, yes, in any field you have people that miss the science for the method. Sometimes that makes sense if their research is on the method and not the application, but the best papers are strong in both.

On point 1, I feel like any kind of statistical cookbook that says “if x do y” will lead to the same problem. Any such recipe will not be able to capture the nuances of problems many scientists face. I’m not sure what the solution is other than some vague “we need better modeling tools” statement that is as wishful as asking people to learn stats better.

On point 2, I also agree that’s an issue. In my field, the “future directions,” section is the first thing you cut down to make a page limit, but it’s really an important section to both discuss what the issues with your idea are and how they can be fixed in the future.

One thing we're trying in my program right now is to introduce "Statistical Thinking" as a mandatory class for both wet lab and computational students, to cover the why of statistics and try to teach people how to think about what they're doing, before they hit the "When X, do Y to collect p-value." stage.

Another thought I had is to replace statistics requirements with a rigorous (but not measure-theoretic) probability class or, preferably a semester of both. My engineering program required probability, not statistics, so I never took a formal stats class. However, my probability class was excellent. If I need to figure out a statistical method now, I might not know what method or <letter>-test to use, but I google a description of the problem and am greeted with stats websites and papers that solve various cases of the problem that are totally understandable using undergrad probability.

The intuition for doing so is that, as this article argues, the hard part is in determining the right probability model describing your experiment. Once you have a good model, running the statistical test is easy (whether there’s a tractable algorithm is another question, but if there’s no tractable way to estimate your parameters, I’d argue as a practitioner a model being too difficult computationally is a good heuristic for your experiment not being robust)

The heart of the problem in life science is that "Experiments are always somewhere between too underpowered, too expensive and too unethical."

Consequently, when you actually manage to run one to completion, everbody wants to squeeze every last itty bitty drop of possible juice from it.

So, you will torture the data as far as you can for anything which even approaches significance.

There's really no good way out of this. At the end of the day, the things you can do to humans are simply a lot more limited than the things you can do to inanimate objects.

I think your comment is spot-on, except for one sentence:

> It’s an understandable problem — if you are a biologist, you want to think about biology, not statistics and most biology work doesn’t require math so it appeals to the scientifically-minded that dislike math and programming.

I think you are being polite, but what on earth does scientifically-minded mean when math is elided from it?

When it comes to biology or medicine the information sources tend to be very wishy washy on average, but tend to be illuminated crystal clear in settings of systems biology (which is basically mathematical models in the context of biology), where simplifying assumptions aren't hidden in a wall of text, and the effects of ignoring terms become clear. If you pay attention you can find non-systems biology descriptions of supposedly unexplained phenomena or interactions for further investigation that are entirely explained long before in systems biology texts...

> most biology work doesn’t require math

The proof of the pudding is in the tasting, not the high-school experience of what a field should continue to look like in higher education.

The life sciences is the only field I’ve worked in where there are specific practitioners for statistical analysis.

It really seems weird to me, but perhaps I’m thinking about it backwards and it’s a sign other fields should do this too?

There’s a justifiable stigma in the sciences and engineering about those in other domains who say “I’m not a math person”, yet it seems like it’s OK for people not to know statistics.

I think part of it is also the nature of the research. As an engineer, the use of statistics in papers I’ve written to compare outcomes of the proposed method vs competitors is rudimentary, but that is for a number of reasons:

1. Normally the proposed method exhibits a number of trade offs with other methods. For example, the proposed method might be slower but accommodates a new class of problem existing methods can’t handle.

2. You can’t easily sample problems for comparison from a real world population. Such a data set doesn’t exist. Therefore, even if I was doing very formal hypothesis testing for a given problem, I could still cheat by adjusting the problem specification until my method is better.

3. Collecting data to a statistically significant level can be incredibly time consuming, and that’s just for a specific problem set up.

4. Theoretical analysis is offered to qualitatively, if not quantitatively compare methods. The experimental data serves more as a sanity check for the theory and demonstration that the proposed method is practical to implement and gives sane results.

I’m sure in the harder sciences, this is somewhat similar. It also helps that you have a lot of control over experimental design so that the statistics you need to do can be simple by construction. In biology, you have a million compounding factors and little theory to fall back on, so the statistics will be more complicated.

As a software engineer with no research experience, I wonder if one answer may lie in more foolproofing of the most commonly statistical analysis software, like matlab and stata. They are hugely powerful tools but obviously require quite a bit of domain knowledge to use properly. Could many of the most common types of analyses be made into more easily runnable happy paths that also warn against common invalid conclusions and methods?

I don’t mean to imply any of this is easy, but I’ve seen other pro-level software successfully integrate happy paths for novices.

There are a number of statistical software packages that have, as you refer to them, "easily runnable happy paths".

I'd assert that these are the source of many of the problems, because statistical reasoning is not purely algorithmic.

No bluescreen level error when I click run?

Resulting answer must be be correct.

I think better tooling may help, but part of the issue is that the tools are asked to do the wrong thing because you’re thinking about the problem wrong. In the article, they cite the hot hand research: for a long time, the understanding was that the hot hand is a myth, but recent work identified a source of significant statistical bias in the prevailing methodology, and identified that the hot hand is real when you correct for that bias.

Better tools definitely will help practitioners, but no tool will save you if you call upon it to do something that isn’t rigorous.

As an aside, I’ve wondered about a system that augments data with units dynamically. You tell it what units your data is in when each variable is created, and it carries that through basic computations. Thus if you divide a “meters” variable by a “seconds” variable and print the result, the result will be in meters per second. The value being that it is instantly obvious if you are doing something wrong because you’ll get nonsense units.

"I’ve wondered about a system that augments data with units dynamically"

F#'s units of measure do this! Though the compiler checks it, so it's static, not dynamic.

https://fsharpforfunandprofit.com/posts/units-of-measure/

If someone followed the "hot hand" resolution link: https://theconversation.com/momentum-isnt-magic-vindicating-...

If this piques your interest I can highly recommend the book:

Runs and Scans with Applications, by N. Balakrishnan and Markos V. Koutras

Reply on news.ycombinator.com