Hacker News Reader

Comment by majidazimi | Hacker News Reader

The problem I can see right now with scientific community is that, no one pays you to re-implement the same idea just to confirm that it is correct. Every professor wants shiny new innovation from her Ph.D student. No one wants you to experiment the currently published ideas.

I'm about to finish my master thesis. I implemented a couple of ideas collected from multiple papers in my thesis, and I can say all of those absolutely stunning results were just intelligently crafted experiments that are not applicable elsewhere.

Basically, my whole thesis is just to show that some paper was wrong and the idea is not applicable in another experiment.

View on HN

I often have my PhD students re-implement ideas to gain skills and to verify a method works. It is possible to publish these efforts, but it isn't easy. It typically involves comparing multiple methods on datasets they haven't been tested on before to see how well the results generalize beyond the original paper. It is hard to publish in prestigious venues with this approach, but we have had some success. Replication and comarison makes for a good MS or early PhD project, but later PhD students have to showcase an ability to create new algorithms. In my lab, we try very hard to not cherry pick and to do good work that generalizes.

Agreed, reimplementation is pretty good for students to learn. Our lab requires everybody to work on an implementation project

From your profile, you are working in Deep Learning. In other fields it's much more difficult to be sure that the replication is accurate.

Do you have the correct variety of rats? Are they receiving the same kind of food? Does they get the same illumination during the day? ... Theoretically al the details should be clear from the published paper, but most of the times the paper is full of "underspecified"[0] parts.

Also, in many fields even if the paper is only about calculations in a computer, the programs are not published (or are a mess (or an unpublished mess)) and the data are not published (or are a mess (or an unpublished mess)). So it's more difficult to even make a direct copy of the results of the paper.

Also, there is a lot of informal replication, essentially what you do but without the final publication step. Just replicate somewhat similar to the original paper, but then publish a version with some extension or tweak.

[0] aka "missing"

> Replication and comparison makes for a good MS or early PhD project

Do you think that might be the solution? That is, to get an MS your final project has to be an attempt at replication, and then a PhD has to be a new contribution. If that became the standard, would it solve a large part of this whole issue?

There is a huge difference between physical and computer sciences. I agree that it's an excellent use of time for new students if the main factor for the work is time/salary. This paper was aimed towards medical/bio/clinicians and when you add material costs, the variability of biology and multiple people required to run large experiments everything falls apart. One of my PhD projects (Primate neuroscience and new medical devices) took 4 years2 grad students2 staff6 animals (only 2 made the paper)animal housing costs for 4 years + ~$150k in materials. And there are only ~10 research labs in the world that have the ability to do this type of research so no one is going to . We see a lot more replication via extension - "this theory worked for this group, what if we take that as true and extend it in a new direction, where does the science fall apart there?" I'm not arguing that replication isn't important and a lot of false positives get through into the literature but without 10x-ing the research budget and infrastructure as well as changing the publication incentives there isn't going to be any real movement.

Yes, that’s true. It’s also because the view is: published == valid.

An idea I had once was to say: the group that repeats or falsifies the results of a paper will be published in the same journal. So this motivate people to piggy back on high profile publications. Unsure if it’ll work though

The idea that just because it was published makes it valid is insane on so many levels. Usually, it is required - or at least good sense - to address limitations and draw conclusions based on your findings.

This assertion flies in the face of that practice.

To be fair, the professors are like this because the funding agencies---ie, the government and a handful of private groups---essentially refuse to fund duplication research. I thus think that the issue has broader origins than the attitudes of individual faculty.

The review panels that score grants at the National Institutes of Health (NIH) are comprised of professors. (Mostly; being a professor isn't technically required.) If a proposal is scored well by the review panel, it is almost always funded. The National Science Foundation (NSF) works similarly. If (a) the review panels were on board with duplication and (b) people started submitting proposals for duplication research, those proposals would probably get funded.

Edit to clarify: The point is that professors, as a group, already have the power to allocate funds to duplication research if they wished to do so. Blame for a lack of duplication research cannot be pinned solely on the funding agencies.

The first paragraph reads a bit like a possibly faulty generalization based on your personal experience as covered by the rest of the post. I might be wrong, but I don't think it's that bad. Depends on the field maybe, and I don't read enough papers myself to have a proper complete view on this, but I definitely recall reading papers which where about recreating another experiment, figuring out that it was correct and adding some more steps to gain additional insight.

I think it's pretty common to ask PhD students to reproduce some results as part of their initial literature review.

Not surprised to hear this at all. I can remember being specifically told during both undergrad and grad research seminars 'It is highly unlikely that you will find a large number of statistically significant results. The sample sizes are just too small'. In social sciences, this is an astoundingly common problem.

Then, of course, when I have to defend my comp. exam (for a second time) in front of the faculty, I spent the last 5 minutes lamenting the very phenomenon you described. Research can be deeply engrossing, but it doesn't mean shit when you go out into the world looking to land your next job. Of my 3 reviewers, only 1 had spent significant time in industry before teaching so, think of this as a wake up call for people to get outside the bubble and broaden their horizons once in a while.

Tl;dr The program I was (Communications) in changed their requirements right after I left to eliminate this distinction and now students are required to develop and implement an idea (Marketing campaign, Training seminar, Comms. Audit w/ recommendations, etc.).

You might get, 'that's nice' or even 'interesting', but this is usually followed up with 'how can you help me solve problems x and y right now on deadline?'

>"'It is highly unlikely that you will find a large number of statistically significant results. The sample sizes are just too small'."

This is just a tautology. In social science large sample size = statistical significance (there are only false negatives since the null model being tested is always wrong).

If this is true, how does that work if the fundamental conclusion re the relationship of 2 entities is the inverse of what you expect? To me, that result doesn't call for adding participants to get to an acceptable p value, it means reevaluate your assumptions, because your basis of understanding is wrong.

@epistasis' using the term 'p-hack' is incredibly apt in this case.

The ability for a study to be deemed "correct" in proving a hypothesis might be true of small samples if you are sufficiently rigorous in designing the experiment, but it could still fall apart at larger sizes - which only proves that part of your hypothesis was not statistically meaningful regardless of the conclusion you reached.

Again, a thorough discussion of limitations around any research should address this.

E.g, "Our findings indicate x, but there is no way to know without further study if y is actually true instead"

>"If this is true, how does that work if the fundamental conclusion re the relationship of 2 entities is the inverse of what you expect?"

Not quite sure what you are trying to say but this doesnt sound like any statistical model I've seen used in social science.

Usually they assume some distribution (eg normal, or "t") and then assume both samples are taken from that distribution. From that they derive the proportion of times you would see the observed difference in means between samples.

It's really the exceptional case I'm thinking of here.

Basically, you screw up such that the data you've collected is meaningless. Sure, you could do the statistical analysis correctly and get a result, but it ends up being the opposite not because the relationship is actually different, but because some part of your design is flawed. In that case, you would never end up with a correct result.

(This is of course the worst possible outcome, but I would be remiss if I didn't mention it here.)

Although I have no background in it, I suppose the same could happen in the other sciences if there are side effects that you don't anticipate and don't measure accurately enough.

^ This again is why reproducibility is important. Earlier in this thread, @majidazmi mentioned that a major focus of their thesis was to prove that a prior experiment was incorrect in its findings.

And me thinking that science isn't about trying to confirm findings, but trying to disprove them and if nobody can do that, the original hypothesis must be true.

Why not arxiv?

Why not arXiv... what?

Why not arXiv pay people to do reproductions? They don't have anything like the funding required to do that.

Why not host reproductions in arXiv to build up a name, similar to how devs use github to build a profile.

The problem isn't literally hosting the reproduction study result PDF files when you're done - you could do that on GitHub or anywhere else nobody cares.

The problem is funding the time and resources needed to do the reproduction studies themselves.

> The problem I can see right now with scientific community is that, no one pays you to re-implement the same idea just to confirm that it is correct.

Reply on news.ycombinator.com