Why we should love null results | Hacker News Reader

Why we should love null results

the100.ci

83 pointsDanBC26 comments

I've always wondered why people call it a “failed experiment” if it disproves the hypothesis. Disproving the hypothesis is just as much an experiment's job as proving it. Doing neither would be actual failure.

Experimentation is really the heart of scientific progress. Yes, experiments will yield results that do not confirm the hypothesis, that will happen very frequently, and that is good because we can chuck unnecessary results away, focusing on what fits the picture accurately, what doesn't, and what pictures we can make. I don't think it's really possible to have failure though, since you would just end up with inconclusive results and do more science.

A "failed" experiment does not "disprove" a hypothesis, it only provides evidence which is not consistent with the hypothesis.

Actually not even that, it provides evidence which is consistent with the null hypothesis.

Experiments disproving the hypothesis are usually deeply unsettling, because in most cases, we have a pretty good reason for having that hypothesis to begin with---often lots of other experiments from the giants in your field. Those really smart guys (and, unfortunately it is mostly guys and not gals) are more impressive than you, J Random Gradstudent, so maybe you just screwed up the experiment. And even if the ruling hypothesis is wrong, then that means you're going to have to come up with a better one that fits both the old experiments and your new one, which is probably going to be hard to do if you want your new hypothesis to be simple and clear.

So a really clear rejection of the alternative hypothesis is great (which the article advocates for), but most of the time it's the more ambiguous non-rejection of the null (as you put it, actual failure), with a slim chance of bashing the old guard with your probably janky new hypothesis that explains it. They laughed at Galileo, they laughed at Einstein, but they also laughed at Bozo the Clown... and you don't want to be a Bozo.

I'm an ESL speaker, but after some searching on the Internet, I've convinced myself that "guys" is generally accepted as a mixed gender group.

As an American speaker who grew up in the south and lives in the Midwest, that's not true in every case. When addressing a mixed group, "hey guys" can usually work. When talking about a group of unspecific people, it can work, but mostly because our cultural default is to assume male. When talking about a group of specific individuals "the guys in the office are smart" I would take that to be more specifically male. It's definitely not unambiguously mixed gender. In the case of the GP, the word was used to make a point about the ambiguity of the word, in a way that's very naturally idiomatic for a native English speaker, imo.

As another native speaker I can say that "guys" refers to a mixed gender group perfectly well. Saying that it implies a predominantly male group is more indicative of your personal life experiences or bias.

Regarding GP, my only guess is that they were virtue signaling because they explicitly went out of their way to make that point. If they really wanted to avoid gendered nouns they could have written "folks", "people", or any number of similar words (kind of like how I'm doing here, by not assuming GP's gender).

'Folk' is fine, though try to work out how to drop 'comrade' into your vocab.

In this particular case, the OP could have used 'researchers' and: 1. avoided putting off/distracting people by the use of gendered language; 2. been more precise.

Also a native speaker, here.

Seems to me that "guys" can refer to a group of men, or to a mixed-gender group, but not to a group of women. So while it's absolutely the case that it can refer to a mixed-gender group, using it that way is still problematic in that it emphasizes men's membership.

I use "guys" to refer to groups of women all the time. I don't think people who say "you guys" check and make sure it isn't 100% women before saying it.

It depends where you are. Some people do strongly see it as not gender-neutral, and will loudly criticise you for using it even if you mean it in a gender-neutral way. Which is a shame, because while language is important there are probably other bits of language that are worse.

Ireland has "hey lads" as a gender-neutral term which surprised me the first few times I heard it.

> Those really smart guys (and, unfortunately it is mostly guys and not gals)

To be fair, I'm not sure why the original poster included the gender here instead of saying "smart people." The gender thing is a bit off topic.

You are right that "guys" is often accepted ...at least as an informal second person plural. I get the impression this is because English just doesn't have a lot of great options here. Personally, I use "y'all." No matter the company you keep (with the possible exception of a group of CEOs) it works and doesn't make things awkward. That said, contractions like "Y'all'd've" start to get a bit messy and I generally stay away from those ...especially when writing.

What's wrong with the second person plural subjunctive past perfect?

"How to derail a thread 101"

Don't disrespect the clowns.

Those really smart guys (and, unfortunately it is mostly guys and not gals) are more impressive than you, J Random Gradstudent

Bigtime survivor bias.

Am I wrong to think anything that shrinks the search space is good?

> They laughed at Galileo, they laughed at Einstein, but they also laughed at Bozo the Clown... and you don't want to be a Bozo.

P.S. Sez Carl Sagan (more or less).

Can you give an example?

I have never seen people say it was a failed experiment when it "disproved" a hypothesis. If it was able to disprove something, often that means people perceive a high degree of certainty in the results.

They might be bummed out and try to suppress its publication because they don't consider it disproof of a favorite theory, though (for whatever reason, good or bad).

Edit: I just remembered that one time I saw people complain that their analyses had "too much power", which might fit in with the parent comment's point?

The matter is that the hypothesis is not disproved. From a statistical point of view, H_0 not rejected just means that we can not exclude that the difference (or the correlation) between two or more variables would be attributed to the chance (that is, the sample variance). The outcome is, of course, informative. If we organize many well designed experiments, and none of them is able to confirm the hypothesis, we can reasonably conclude that it can't be proved. The null hypothesis, however, is not as strong (in disproving a theory) as the alternative hypothesis is in proving it.

they failed to disprove the null hypothesis, but that does not disprove the alternative hypothesis

Someone with enough social capital should found a journal dedicated to only publishing null results. It could start out multi-disciplinary, and spilt up if the publication load got too high.

Everyone could setup a digital journal on an afternoon, but it had to have the blessing of people with reach in the scientific community to get anywhere.

It already exists for some fields, but these journals tend to be cited less, which hurts their scores in impact analysis. The problem goes deeper even than publication bias.

http://www.jasnh.com/

A lot of academic research is no longer fundamental and is essentially r&D, note the big D. Funding sources such as the EU commission are mostly interested in funding what will have near term economic impact. So you've got entire research institutions who have nicely kitted out clean rooms and all kinds of cool toys doing just enough on each project to keep the funding coming in.

As a result negative results will piss off project coordinators so who want the next big thing they've been promised and research has just become complex assembly projects between collaborators where researchers repeat what they know works.

In the current culture for Science we emphasize new or interesing discoveries and publish or perish. Therefore writing up the null result is time consuming and doesn't benefit the researcher . I think if we had a easy to use way to collect null results and indexed them to other researchers it might make a difference . To be honest this solution would also be a hard sell.

Reply on news.ycombinator.com