Hacker News Reader

Comment by infogulch | Hacker News Reader

I like the idea of splitting "peer review" into two, and then having a citation threshold standard where a field agrees that a paper should be replicated after a certain number of citations. And journals should have a dedicated section for attempted replications.

1. Rebrand peer review as a "readability review" which is what reviewers tend to focus on today.

2. A "replicability statement", a separately published document where reviewers push authors to go into detail about the methodology and strategy used to perform the experiments, including specifics that someone outside of their specialty may not know. Credit NalNezumi ITT

View on HN

Every experimental paper I've ever read has contained an "Experimental" section, where they provide the details on how they did it. Those sections tend to be general enough, albeit concise.

In some fields, aside from specialized knowledge, good experimental work requires what we call "hands." For instance, handling air sensitive compounds, or anything in a condensed or crystalline state. In my thesis experiment, some of the equipment was hand made, by me.

Sometimes specialized facilities are needed. My doctoral thesis project used roughly 1/2 million dollars of gear, and some of the equipment that I used was obsolete and unavailable by the time I finished.

> My doctoral thesis project used roughly 1/2 million dollars of gear,

Wow I envy you. My doctoral thesis project spent like... USD2.5k directly for gears (half of it just to buy lego bricks to build our own instrument exactly because we can't afford to buy commercial one lol)

I used a 3 billion dollar space telescope. I don't think NASA are going to launch another to replicate some of my results.

“Concise” isn’t good enough. If other scientists are trying to read through the tea leaves at what you’re trying to say you did, that defeats the entire point of a paper. The purpose of science is to create knowledge that other people can use and if people can’t replicate your work that’s not science.

I think the point is you don't have to give a complete BOM that includes where you got the power cables. Each scientist has to decide what amount of information needs to be conveyed. Of course this can be abused, or done sloppily, like anything else.

A place where you can spread out more is in dissertations. Mine contained an entire chapter on the experiment, another on the analysis, and appendices full of source code, schematics, etc. I happily sent out copies, at my expense. My setup was replicated roughly 3 times.

Yes, maybe there should be a process where peer scientists review a paper before it’s published, to make sure it is written clearly enough for other scientists to understand it.

Reply on news.ycombinator.com