Hacker News Reader

Comment by nonbel | Hacker News Reader

We will never be able to enforce this and do productive research. It is all due to mathematicians' fantasy about how research works and researchers' fantasy about how stats work.

Something always needs to be adjusted that is figured out during the study. Focus on testing precise predictions and observing similar results in different studies rather than significant differences. Despite it being much his "fault", Fisher pointed this out nearly 60 years ago:

Fisher, R N (1958). "The Nature of Probability". Centennial Review 2: 261–274

View on HN

The standard that COMPare is aiming to achieve, as far as I understand it, is the following:

1. Outcomes that will be measured for clinical trials need to be registered and public ahead of the trial (this is already achieved with recent regulation, as mentioned in the article)

2. If planned outcomes change, the change needs to be reported and explained, with new outcomes being again clearly stated.

I really struggle to see how enforcing this will prevent productive research. This is basic scientific hygiene, not fantasy. No one is arguing that studies shouldn't be adjusted, just that the adjustments should be clearly stated.

Completely changing the goals of a study after failing on your initial goals, and then claiming success without mentioning any of the failures along the way is poor science, because you're masking a lot of potentially significant findings along the way that others can benefit from as well.

Not to mention that these standards have been agreed to by all the largest journals decades ago, with big PR claims of how this is needed etc., yet are never followed in practice. The responses of some journals (NEJM, JAMA that I recall) have been shameful, and contradictory to standards they claim to follow.

Misrepresenting your methods is just fraud, so you will get no argument from me there.

However, think about the difference between A) setting a point prediction of your model as the null hypothesis vs B) "two groups are the same and samples were independent, etc" as the null hypothesis.

In the first case there is a very small range of plausible outcomes consistent with the researcher's theory. In the second case there is a very large range (usually 50%). In the case of scenario A, messing up the experiment (either on purpose, or accident) makes it more difficult to claim evidence consistent with your model. In the case of scenario B, the same thing makes this easier.

So researchers using scenario A are incentivized to be as careful and account for as many different sources of error as possible. Under scenario B, the incentive is the opposite, the sloppier the study the easier it is to get results consistent with the researcher's theory.

Do you think Study 329 was productive research? (Productive for patients, that is, as was productive for GSK's income.)

If a study is always adjusted then how can you make a precise prediction about it?

Once you've done the research and found a post-hoc correlation, what additional tests are needed before making a policy decision, like recommending Paxil for depressed youngsters?

If there was outcome switching, is it okay to omit noting that in the paper?

Sure we can. Wall St does their best to enforce this because the alternative is losing money. Models work great in testing but fail horribly in the market.

Academic scientists feel that they can't be "productive" with rigorous stats because they consider "deploy the model in published paper form" rather than "see if it makes money" to be their end state.

It is true that in practice, you can't foresee everything that is going to happen in a practical experiment (you wouldn't do it anyway if it was the case). But it doesn't justify not enforcing clean research practice. If you need to move the goalposts, you can do it, it's call "adjusting the hypothesis". You make a new hypothesis and you test it properly. Sometimes you need to iterate a lot, and you fall into the risk of https://xkcd.com/882/. Which still underlines the importance of having a strict scientific approach.

Reply on news.ycombinator.com