Hacker News Reader

Comment by chalst | Hacker News Reader

Andrew Gelman had an interesting 9-years-too-late idea for what to do in cases like Bem's article:

https://statmodeling.stat.columbia.edu/2019/01/27/jpsp-done-...

Rather than publish all of the dubious speculations and the analysis whose problems took time to emerge, simply publish the data. The advantage, in his words:

> In the example of the ESP study, if anything’s valuable it’s the data. Publishing the data would get the journal off the hook regarding fairness, open-mindedness, and not missing a scoop, while enabling others to move on reanalyses right away, and without saddling the journal with an embarrassing endorsement of a weak theory that, it turns out, was not really supported by data at all.

View on HN

Incentives: Spend five years doing difficult / expensive / dangerous research, gathering data that demonstrates X. The data set is vast, and you can get ten papers out of it, one every six months with some awesome collaboration and possibly tenure.

Do you do that or

publish all your data in the first paper and watch a dozen grad students publish your next nine papers in three weeks ?

Incentives matter yes. Science and data should be free and transparent. But if we pay peanuts and expect the monkeys to appreciate the applause of publication, we need to change our incentive structure.

The reasons AI / ML researchers don't mind publishing the data early is because they already get 500k salaries and equity.

Then there is a huge problem with how we publish Science... Decades ago, sharing all these huge datasets would have been quite difficult if not uneconomical. But nowadays there is no reason left for not publishing the data and hopefully get academic rewards for that... I have seen huge amounts of data not being digitized and lost because of this ownership mentality...

It is not a 'mentality' issue. It's a systemic issue. Researchers need to eat too, and it's already difficult enough that few are willing to sacrifice much for the sake of morals.

publish all your data in the first paper and watch a dozen grad students publish your next nine papers in three weeks ?

The way I would expect this to work is, the grad students give you credit for gathering the data, which allows you to share in the prestige associated with all of their projects.

No?

Alas, Kepler is better known than Brahe... Acknowledgements don't fix the incentive problem.

That's like someone releasing a CGI TV series called "Woody and Buzz in the toy store" with exact copies of the digital models. And not expecting Disney to say "we spent 1 BN promoting this brand, we expect a cut / control / story approval / your heart on a plate " [#]

[#] I believe that last one is in actual fact the Disney legal teams' opening gambit in all copyright negotiations. Then the gloves come off.

If graduate students can publish papers based on your data three weeks after they first see your data, then surely you are slacking off if you can only publish one paper every 6 months.

More plausibly, if your data is that apparent,

- you publish your first paper with your data, this takes 6 months from submission to publication

- you submit a paper every three weeks (also with your data) during that 6 months. that totals 8 papers before your article and data hit the press

- after your paper is published, it is 6 weeks before anybody fully gets the data

- during that time you submit your other two papers

- the grad students you mentioned now submit 8 papers, but are forced to cite your work

You now have 10 papers submitted before the competition and have 10-20 papers being submitted that cites your data and work. Total elapsed time = 14 months.

Sounds much better than your 5 year scheme.

sorry, citing by other people do not have value equal to publish article on first author... and just do experiment should spend very very time and money, e.g., "if I add this compoment to cultured cells" just spend one day and then spend half of year to add component to cell each day and collect results...

You seem to be forgetting that many western groups are very small: 1 senior (often not even tenured) and 1-2 grad students, with not even a lab manager in sight.

In those conditions, going slower than a 50-grad-student chinese lab is not slacking off.

Obvious question then: what are the arguments/forces opposing this? Just inertia in the publishing norms?

The approach fails miserably when taken in aggregate. I would read five to ten papers from my own field a week while in grad school. If I had to do the analysis for myself as well there wouldn't be any time to do my own research.

Add on twenty more papers a week from adjacent fields where I don't even know all the techniques and the magnitude of the problem hopefully becomes clearer.

The raw figures would be available in addition to the paper itself, not instead of it.

Gelman's suggestion is how to handle articles with unlikely conclusions: don't publish the submitted article, though that presumably would be circulated as a preprint, but just the data. Then the journal is bringing attention to interesting data without endorsing the dubious analysis or conclusions.

Thanks, must have read that too quickly :-P

It doesn't strike me as as very practical step. Pre-registration seems much more sensible.

There is an incentive against doing it. No researcher wants the embarrassment of having an error that they didn't know about discovered. Journals don't fancy the embarrassment or hassle either. It would serve broader scientific progress, but that's not who is deciding to hide the data.

A tragedy of the commons, then?

If you ever find yourself thinking like this, get the f out of science asap, you have no idea what you are doing.

It follows from your comment that we should have anonymous authors in addition to anonymous reviewers. Can a researcher establish a provable track record of doing quality work while remaining anonymous?

only if they are independently wealthy

That would never happen, again because of incentives. Academics want to build their brand and publishing anonymously doesn't help with that.

Presently, for sure. I was wondering in the hypothetical, too, though. Authors really just need to provably show that they have a strong influence/presence on the publication graph or something. Although, reducing a researcher's entire value to a single scalar index is probably fraught. Also, egos would never go for completely anonymous authorship system, I think.

There's likely multiple forces, but at least one must be money. Someone's gotta host the data, right? That falls on either the universities or the publishers.

Reply on news.ycombinator.com