Hacker News Reader

Comment by tenaciousDaniel | Hacker News Reader

I'm not a scientist, but this is very surprising to me. I would have imagined that our scientific culture would be one where any result is recognized as something from which valuable knowledge can be gained. The idea that unanticipated results reflects some failure on the part of the researcher seems so strange and unscientific. But again, I'm not a scientist.

View on HN

In an ideal world, scientists would be judged solely on their study design and execution without considering the outcome. But that takes a lot of effort. It's much easier to judge based on the results that the scientist achieved ("I discovered the priming effect"). We remember Watson and Crick, and forget many other scientists who worked just as hard.

Science doesn't really have a place for participation trophies. That and it's pretty natural to judge based on results. When handing out recognition and fame, the results are probably the best criteria - though that should include those who helped and that each stood on the shoulders of giants. That information should be included in the papers for those who are curious.

Negative results are real science, and science is collaborative. When we don't incentivize negative results, scientists don't take the time to analyze and share those findings, which results in a shallower understanding and possibly wasted effort by other scientists. It also biases scientists towards claiming results where none exist (e.g. p-hacking).

The solution isn't publishing negative results. The solution is publishing better results. The solution is holding people to a higher standard and insisting on reproducibility before accepting the findings as factual.

There are times when negative results pose new problems or offer new questions or methods. Those should be published but don't normally get you much praise. A blanket statement about publishing negative results is just going to get overwhelming information that nobody has time or interest in.

Frankly, there are more productive things that occupy a researcher's time. If you want to know what a researcher has tried and failed, then chances are you're a professional in the domain. You should already know how to contact them and you can ask them to share their journals and logs with you.

But, nobody is going to submit all their negative results. Nobody is going to index, catalogue, cross-reference, fund, and disseminate these massive negative results.

If you want to know all the things I've tried and failed (I am a retired scientist), it would take multiple volumes and far more time than I'm willng to devote. Prior to retirement, I had positive results to publish, more things to try, and to find a way to make it commercially viable.

Those acts are far more valuable than publishing things that got a negative result. Publishing those would take all my time and I'd never have gotten anything done.

The system works. The system was designed as it was for a reason. Yes, some negative results should be published but they are exceedingly rare. The problem is a lack of effort to reproduce prior to acceptance and low-quality peer review. The solution isn't to publish more stuff, it's to publish less and to ensure the quality is there. The solution probably should also include open publishing. Instead of locking it up in expensive journals, make it more open so that more people, even layman, have the chance to review and replicate it.

Speaking of which, I mention again that I lament the lack of emphasis, appreciation, and support for the citizen scientist. I'm a scientist. I'm nothing special. I'm wrong more times, by 09:00, than most people will be all day. Stop putting scientists on a pedestal or in ivory towers.

You really don't want our many, many volumes of negative results. Just because it's got a Ph.D. tacked onto the end of the author's name doesn't mean it is valuable.

Really, just uphold the standards that many seem to have forgotten. You're correct in that some negative results should be published, but they are few and far between. If anything, we have too much being published and a distinct lack of quality.

I will add this...

Journals only want a trimmed down paper. A journal may want you to trim your paper down to 10 pages, some want even shorter. Your research may be 1,000 to 2,000 pages but only a tiny bit gets published, or even submitted. That full research will have many of the negative results, as may the 'private' journal that the researcher worked from. If you want that information, you can probably get it emailed to you. The institution may have it in their archives, if the researcher is no longer available.

So, negative results are out there, but most don't really need to be published. Most researchers don't have time, resources, or inclination to publish negative results with dubious value. Sometimes a negative result is potentially valuable, as it introduces something novel. Those get published with some frequency, though rating it as novel and anticipating its ability to further the arts is easier said than done. Even still, those rarities are even more rarely deserving of accolades. There are no participation trophies in science. That sort of thinking is kind of how we got into this mess. Somewhere along the lines, someone should have said, "No, that's garbage. Don't publish it under the heading of science or with the prestige of this journal/institution attached."

There was no relationship between variables is not participation trophy. It is a result and should count as one. As long as science is concerned with finding truth.

If you want to see the negative results, contact the researcher. There is no room, publishing mechanism, or benefit to most negative results. If the negative results are novel or reveal a new method, they will get published.

Science doesn't care that someone tried hard. Science isn't about applauding effort but is about results. Negative results with significance may, very seldom, be worthy of publishing.

Publishing the vast majority of negative results would be a waste of time, funding, and resources. Maybe you're unaware of the many failed experiments across the domains, but they are plentiful.

If you want to read them, the data is still collected - it's just not published. There's no good reason for it to be published in all but the rarest of cases. Just email the researcher, it will be in their journal. If they published, it's in the stack of notes and was trimmed out because it takes space, time, and money while having negligible value.

It's great that someone put a lot of effort in, science does not care. Frankly, we are publishing too much junk as it is. Adding to the list of junk isn't going to solve any problems, it's just going to create problems.

The solution is to increase the quality if what is being published, not to increase the noise to appease some emotions. Again, there is no participation trophy - nor should there be.

What if there were an open platform where these results could be published? Not only published, but the data related to variables could be fed into a database. So if you wanted to check which variables have been found to not be correlated, you could query the database. It seems like that could be useful for scientists who want to avoid wasting their time.

Underpowered negative results are a class of potentially genuinely uninteresting results.

Interestingly, underpowered positive results are likely to be published, but unreproducible. That is, the ratio of false positives will be higher among underpowered studies. So, the literature of underpowered studies will be enriched for false positives.

So, underpowered results, whether positive or negative, are less interesting.

Most ideas people have suck. That's sort of the problem of new ideas in particular. When you're working, their failures aren't recognized as some kind of thing you need to tell the world about, you just tried something speculative and it didn't work out. Writing papers sucks (though it can be ego satisfying if you did something really cool), and writing papers about negative results that no one will publish sucks even more (or give you money for). Also, if you do something really dumb, now everyone will know about it.

Yet people still do studies on alternative "medicines" and other stupidity like "wind turbine sickness." That is literally doing something dumb that everyone will hear about.

Doing studies on those things you think are stupid is the right thing to do. Counter-intuitive stuff is found all the time in science. Occasionally those alternative medicines get tested and turn out to work, in which case they become regular medicine.

By declaring something as "too dumb to study", you're a priori declaring the result, which is bad science.

For what it's worth, I'm just describing the ground reality of research, not the utopian vision we should aspire to.

Which is funny, because in science we narrow down answers by crossing ideas off a list. Not so much proving an answer as disproving others. But you don't get funding, especially in the softer fields, when you don't produce positive results. Unfortunately this also sows distrust in other fields.

Reply on news.ycombinator.com