I wonder how many times similar "mistakes" have been made by PhD students across disciplines.
I wonder how many times similar "mistakes" have been made by PhD students across disciplines.
It was frustrating to hear them voice their opinions in the defense that they felt, “of course it world work.” After seeing the data, they took the exact opposite side claiming it was obvious to the point of being of limited publishing value.
Also within academia there is still a wide spectrum of intellectual rigor across the disciplines. Some things are just more verifiable than others.
The fact that the two interactions were very different with four years and a completed thesis between them doesn't surprise me at all. My own embarrassing story is that I advised Jason Donenfeld to submit his WireGuard paper to NDSS, forgot about the meeting entirely after a few months, then complained (in retrospect, unfairly) when NDSS accepted it. Advisors do stupid, embarrassing, forgetful things all the time. The OP's story isn't even a misdemeanor.
As I recall, he restructured his lectures, asking upfront for a show of hands as to which outcome everyone anticipated, before the big reveal. After making this change, he had fewer people approaching him after lectures saying how obvious the outcome was.
• Predictably Irrational
• The (Honest) Truth About Dishonesty
• The Upside of Irrationality
I really had no idea about this one, but how it's studied is very interesting.
https://www.livescience.com/18101-infants-grasp-gravity.html
I've been trying to get better at recognizing the bias and switching viewpoints without the external push.
When I'm stuck, or getting close to stuck, I always try to assume what I want to do has already been done in some way. Long Google searches or discussions with domain experts, purposely vague, looking for similar ideas. Even a ridiculously not-so-related paper or mention in a paper will launch me in a idea-generation frenzy and I'll quickly build confidence.
Love M. Abrash's books. Shame he didn't keep writing them, they were inspirational for me.
I suppose the process of acceptance will pass through the usual four stages:
(i) This is worthless nonsense;
(ii) This is an interesting, but perverse, point of view;
(iii) This is true, but quite unimportant;
(iv) I always said so
See also the "file-drawer problem" (https://en.wikipedia.org/wiki/Publication_bias). Also, with regards to the incentives in the field and the lack of null results, there's always Ioannidis's classic work (https://journals.plos.org/plosmedicine/article?id=10.1371/jo...).
I've found that looking at what a paper doesn't report can be far more important that what they claim.
I think paper discovery has recently suffered a huge boon thanks to ConnectedPapers, though. [https://www.connectedpapers.com/]
A null result simply means you tried something and it didn't work. But you don't know why. You haven't proven it didn't work. There are literally millions of reasons why something might not work. For instance, you could try to use compound X to cure disease Y, observe no effect, and conclude that X doesn't cure Y. But what if somewhere in the process of making X you made an uncaught mistake and you instead used X'?
A negative result means that you tried something and you came to the proven conclusion it doesn't work. This is, crucially, as hard to obtain as a positive result. In my example, it would imply a much longer process than simply "apply X, see no effect in Y, make a few robustness checks, done".
You could say "Well, publish the null anyway, somebody will catch the mistake". Unlikely. There are already so many papers out there that keeping up is impossible. If we were publishing also null results this number will grow tenfold at the very least. Nobody could possibly check everything. They will see a paper "X doesn't cure Y" and call it knowledge, stifling a possible cure virtually forever.
Am I splitting hairs? Perhaps. But I think HN prizes itself to be a scientifically minded community, and thus it has a mandate to use terms correctly. Confusing "null" with "negative" is a sin.
I hope one day I'll find a way to strongly and passionately argue against the "null results are as important as positive results" position. It is a bad meme. Charitably, I consider it most of the times a honest mistake. But sometimes it gives me the impression it is a cheap trick used by people to erode the reputation of academia.
Correct. There is a highly cited paper in CS where the author showed that a mathematical model that was widely used in research didn't actually work (anymore) in reality. That paper was the starting point of a lot of new research in that field.
Null results are also important.
Suppression of null results allows for p-hacking and confirmation biases to creep into research, and greatly reduces the power of literature reviews.
What I'm arguing against is publishing a null result as a stand-alone publication. This creates the illusion of it being somehow a "result", which is not (in fact, we should stop calling them "results" altogether). With a null you haven't proven anything, and thus it is not a sufficient basis for a publication.
(Of course, ideally I think we'd be better off focusing on reporting the data in a Bayesian approach, but that hasn't really gotten traction in the broader community.)
The difference between a null and a negative is just that a negative is an interesting null. In your null example, to create a proper negative you'd probably report several compound synthesis methods instead of one. You'd probably also want to use more mice/data in your analysis.
I agree they're different but, but disagree that they're worlds apart. There's a spectrum between them, caused by uncertainty and statistics. If I say the average treatment effect of my new drug is probably somewhere between -x and +y, it could be a negative result or a null result. It's the fuzzy line between statistically insignificant and materially insignificant.
Maybe I only had two patients per experimental cell, so I barely learned anything. The drug's treatment effect on lifespan is between -30 years and +10 years. It's "null" in that we didn't learn much of anything.
Maybe I had a billion patients per cell and I learned that the average treatment effect on lifespan is between -0.001 days and +0.1 days. It's "negative" in that we learned the drug doesn't materially affect lifespan.
The position we seem to be in is that most conventional experiments are powered with a moderate effect size at 80%, meaning that many of our null-or-negative (-x, +y) results will be right around the region where it's unclear whether results are null or negative.
Of course, the problem with all of this is that there really aren't very good incentives to accurately and carefully report null experimental results (except as a kind of "folk knowledge" within a given lab) which would limit its general usefulness. But the "platonic ideal," so to speak, of a null result journal I think would be relatively useful.
However, the uninteresting part would definitely help others to not make the same mistake. The uninteresting result is still result (and contribution), isn't it?
You cannot make a TED Talk about something that people already know
This actually sounds like a really good review paper! Review papers serve multiple purposes: getting people up to speed on a subject, and putting your own spin on a subject to guide future investigation.
2. As an aside, I can't tell you how many times I've tried to work on stuff, it ends up working, and then I find papers and people saying what we did would never work. Sometimes the ignorance is good.
Seriously? What kind of scientific paper makes such claims?
There is a range of journals from high ranking to solid mid-level to lower tiers to somewhat suspicious to downright obviously pay-to-publish. You can always find a level that will publish your article.
I know one person who basically couldn't get their PhD because they couldn't reproduce another experiment and after several years of trying is pretty much certain the original results were faked in order to be publishable.