Hacker News Reader

Comment by jvans | Hacker News Reader

in my experience academics are the worst at understanding uncertainty. They refuse to believe anything that doesn't pass their beloved t-test, believing the holy threshold of 0.05 is the arbiter of all truth

View on HN

You are conflating the typical threshold for drugs to pass random placebo controlled trials with all of academia. That demonstrates a lack of understanding and places you firmly in group 2.

It's not just drug trials, it's a huge proportion of biology and psychology research at least. Additionally, most research in biology is expensive to get off the ground, and in order to get funding it absolutely is important to project confidence - there's an entire game surrounding grant writing that sounds a lot more like "group 2" than "group 1". Group 1 is just a theoretical ideal, not the reality of academic science. While that may be somewhat field dependent it's definitely not restricted to a small niche either.

My wife's course exercises in statistics for health studies says "if the p-value is below 0.05, the relationship is statistically significant." Sure, it's a medical institution (Karolinska institutet in Sweden), but the programme she's in is about workplace health management, not even close to drugs.

Two data points do not a generalization make. I work in engineering sciences, and we have no such threshold.

This actually strikes me as an example of a political decision creating a stickiness to academic formal standards (as others point out this has become a policy for various organizations' approval of findings). Since the "replication crisis" that happened in the last decade, a lot of people's explanations for this have centered around the use of the fisherian significance test and how gameable it can be, as well as pressures toward publishing positive results. I think there's some merit to solutions involving changing those standards and a lot of STEM people have advocated for them. However, things like drug trials have politically-determined standards that need to be upheld, and some of those standards have become well-known by the general public, so they can't really be bugfixed quickly

There is no single or set of standards that can prevent you from having to think critically about a result. Rarely are results so overwhelmingly positive that it's obvious what the result means. In those cases it doesn't matter what method you use anyway. The rest of the time, you should use many methods, be wary of their assumptions and think very hard about what the data is telling you. Realize that you are biased in favor of a positive result and play devil's advocate with yourself. This is very very hard work, and most people don't want to do it, especially when their careers improve when they look the other way

I view this as obvious but I guess that doesn't go without saying. The two cultures I've characterized actually share a commonly held strong belief that there exist metrics or heuristics that can replace the need for fluid critical thinking. Obviously it's not everyone in either culture who thinks this way, but it's a dangerous form of automation for both

The problem I have with experts is that they are confident that they understand what someone else is articulating.

One problem is that experts use their own jargon and often confuse one use of normal words with their jargon.

The second step is dismissing rather than inquiring.

Reply on news.ycombinator.com