Hacker News Reader

Comment by cbcoutinho | Hacker News Reader

This has been my experience as well. I'm also am afraid that these researchers will succumb to a similar replication crisis that devastated the social sciences [0]. If you can't necessarily explain why something works, it's probably difficult to explain trying it again in the future didn't work

[0] https://en.m.wikipedia.org/wiki/Replication_crisis

View on HN

It's more like the opposite problem.

Psychology researchers try to justify deep, compelling theories of human behavior with small experiments that are hard to reproduce.

Deep Learning experiments are comparatively easy to run and reproduce because the field has both a common set of performance benchmarks and a large community of people implementing cutting-edge ideas in common frameworks. The harder part is building useful theory on top of those experiments.

This is not really the case. I have personally quite literally implemented papers the week after they appeared on ArXiv and they worked. Anecdotical yes, and there have been papers that nobody was able to replicate, but at large the community is quite good at recognising which work can be used, and this work is then implemented and disseminated.

The situation is slightly better than social sciences though. Social science experiments, if any, can hardly be repeated. AI experiments can be easily redone if the source code and datasets are released into the public.

Reply on news.ycombinator.com