Hacker News Reader

You are reading that completely differently than how I read it. I personally think that the author is reflecting on the fact that he may have been not focused enough on completing a single task before starting another. It sounds (and this is speculative, I don't want to put words in the author's mouth) like the author would start with a hypothesis, run an experiment that generates data that appears to support that hypothesis, but then start running tangential experiments disproving alternative hypotheses to further strengthening the original hypothesis. Hence his refelection:

> I am not advocating the reckless publication of unverified data, or recommending being so careful that you end up publishing nothing — but there is a difference between being reckless and shooting yourself in the foot.

To me, the author is looking back and now realizes that he may have been too cautious in that he was running too many experiments to publish one result and that there is a spectrum of caution and that he may have missed the sweet spot. As others have pointed out ~19 papers (and only 5 first author) over 23 years seems to be a low publication rate, which I think the author is reflecting on.

I agree that the incentives are messed up and these upside down incentives are a driving force behind the reproducibility crisis, but I wouldn't take this quote from the author and say it is an illustration of the reproducibility crisis.

> As others have pointed out ~19 papers (and only 5 first author) over 23 years seems to be a low publication rate, which I think the author is reflecting on.

Low compared to what? Compared to the expectations set by his fellow scientists in a field shaped by the norms and incentives in question.

There's a circularity here (and in the earlier comment by Obi_Juan_Kenobi): apparently this guy's real problem was his own low productivity, which we know because he published less than his colleagues and competitors -- therefore his case can't tell us anything about the warped incentive structure that favours publication maximization over intellectual integrity.

By the author's own admission, he thinks he could have focused himself a little better. He isn't saying he would have had to cut corners and do poor science, just not "shoot himself in the foot" to use his own phrasing. In reflection it sounds like he thinks that there was an opportunity for him to publish more frequently without doing bad science.

This is why I don't read his self-reflection as necessarily being an illustration of the type of mindset that leads to the reproducibility crisis. It sounds like a reasonable scientist reflecting on things he might have reasonably done better with the benefit of hindsight. No where do I read the implication that he is lamenting his choice to not do shoddy/quick science for the sake of securing a tenure track position.

Moreover this is self reflection so a lot the things he is reflecting on are speculative in that he thinks they might have helped him find a tenured position, but he can't be sure. He even has this disclaimer at the beginning of his piece:

> I’m not claiming to know the formula for how to get tenured (if I had that, I would have used it for myself). Instead, I would like to offer the advice I’ve gathered.

Yeah - I know scientists who do this and it has nothing to do with the reproducibility crisis. It's a desire to have a "complete" story, to have explored every nook and cranny, and have the perfect, flawless magnum opus of a paper.

That way lies madness.

What's good for the field (we've decided) is when we only allow a 5% Type I error rate (95% of published positive results are correct). What's good for the individual is to drive their Type I error rate up just enough so that they can publish as much as possible without getting caught.

Umm... That's not actually what only publishing a result when the p-value is less than 0.05 accomplishes. Such a publishing policy instead guarantees (or would if there were no other problems) that __only 5% of the FALSE ideas that researchers test get published as being "confirmed"__ (i.e., when the null hypothesis is true, only 5% of the time does it get "rejected" at the 0.05 level). It guarantees nothing about what fraction of these supposedly-confirmed ideas are actually true. That fraction could vary from zero (if researchers never think of true ideas) to 100% (if researchers never think of false ideas).

You're right, I was trying to dumb down p-values to make a point about perverse incentives in academic publishing. What's worse is that other factors that increase the Type I error rate (only positive results are published) only compound the effect.

A 0.05 p value very much does not mean there's a 95% likelihood of the result being true. It's a fallacy that is far too common.

Like it or not, being a good scientist (read: getting tenure track and funding) is not just about good science, but also exciting, cutting-edge science.

If a scientist runs a couple experiments and find something truly remarkable, the guy who rushes to publish is going to win out over the guy that says “this needs another year of work before it’s perfect”.