Hacker News Reader

Comment by biophysboy | Hacker News Reader

This is a pipe dream of mine, but I would love a Wikipedia of null results: nullpedia! Nobody publishes null results bc they’re not exciting, but I think a lot of NSF money would be saved if “failed” experiments were aggregated somewhere in a searchable way.

There’s lot of questions. How to organize it? How to encourage participation? How to maximize usefulness while at the same time minimizing volunteer effort? How to encourage discussion (suggesting changes for a better exp design) rather than manipulation (stealing the seed of a bad experiment to publish at your better funded lab)?

I don’t know how to do it, but I think if done right people would really like it.

View on HN

I am not sure this would be as useful as the new method of "pre-registered studies" [0], where the null findings appear automatically in the publications. Null findings are not the goal of this approach, but a side effect. Important are the initial hipotheses, which pass peer-review.

[0] https://www.nature.com/articles/d41586-018-07118-1 via https://news.ycombinator.com/item?id=18297724

The Series of Unsurprising Results (SURE) Journal is starting to do this, but only for economics papers: https://blogs.canterbury.ac.nz/surejournal/

Vox recently ran a good profile on them and the importance of null results in general: https://www.vox.com/future-perfect/2019/5/17/18624812/public...

Snopes and Mythbusters showed how you can actually make entertainment out of "This thing we thought might be true, actually isn't true / doesn't work at all".

As for the fairness issue, my first thought is to have it moderated by scientists but from different fields. It's like we do in other areas, like law. You need an expert on the general process, but if they have any personal connection to the topic at hand, they must recuse themselves.

In a sense, it's just a specific type of scientific journal, right? An online journal of only null results.

In a similar vein, I'd love to see some form of reward for replication studies, which are also hard to publish, since they're not novel and exciting. However, due to this focus on novelty, we got plenty of unreproducible results: https://en.wikipedia.org/wiki/Replication_crisis

In my ideal world, a scientific result would not be taken as meaningful until it's been replicated at least once.

Good idea, and great challenges indeed. Taking this further, I'd call it experimenpedia, with experiments published as they are thought out. Be open for comments/review and showing related experiments before the actual experiment being done. This might prevent potential failures and let owners tweak the planned experiment before being actually done. Then do the experiment and publish the results, whatever they are.

I guess secrecy would actually win though, and nobody would use this?

eLife is sorta like this. The experiment isn’t live, but the publishing process is.

https://elifesciences.org/about/peer-review

It’s pretty interesting

To do this, it somehow needs to gain the reputation of a publication but also not encourage people doing just frivolous research (at least not on their own dime).

But I think this kind of thing would be INSANELY useful. Especially if data was attached. (This could possibly help with the above problem because reputations develop).

Great idea, would love to see it.

Related, there were journals like this at some point, e.g. International Journal of Negative & Null Results, or ournal of Articles in Support of the Null Hypothesis, and I think at least one more. Not sure how they fare, though.

Reply on news.ycombinator.com