Hacker News Reader

Comment by bArray | Hacker News Reader

The situation is out of control. I've also seen that during peer review a reviewer may give a list of publications they 'suggest' for the literature review, sometimes completely unrelated fields. If you choose not to, you will likely see your paper rejected.

The truth is that almost all papers published across almost all fields cannot be replicated, full stop. For those that can, the results rarely match up, indicating that the tests are chosen to boost the performance of the method.

If you want to boost citations legitimately, purposefully leak your own paper to get past the publisher pay wall, make it colourful and accessible (if it's too dense they are unlikely to understand it), give examples on how to replicate your method, the abstract is likely the only part that will be read so make it count.

View on HN

> I've also seen that during peer review a reviewer may give a list of publications they 'suggest' for the literature review, sometimes completely unrelated fields. If you choose not to, you will likely see your paper rejected.

You are severely misrepresenting how things work here. A reviewer doesn't accept or reject a paper. A reviewer gives their response to an editor and the editor makes a decision. The editor knows who the reviewer is, and has access to the reviews and the author's response to them.

If the editor sees the reviewer has suggested adding bunch of references to their own work, and has suggested rejecting the paper after the authors refuse this then the editor is going to know what is going on. They aren't completely stupid.

> purposefully leak your own paper to get past the publisher pay wall

Many papers in maths, physics and computer science appear on arXiv before being submitted for publication. In the fields I have worked in you can replace the "most" with "pretty much all".

I think you’re giving the editors too much credit.

I guess it depends on the journal. If you're publishing in some low tier or even predatory journal then yeah there might be problems like this with the editor but in good, respected journals the editors usually do their jobs properly.

> The truth is that almost all papers published across almost all fields cannot be replicated, full stop.

This is not true for Mathematics.

Sure, but most Maths papers are read by like 3 people in total globally, so there's that.

Citation needed. XD

The pareto effect!?

If that's true it's only because most people don't understand these pure maths fields. But their results are generally very useful to the applied mathematicians (whose results are and should be used in most other fields) and in this way there is still an impact although it's not always reflected in the citations.

The fact is that these pure mathematicians could almost surely write the more applied papers but don't because it would feel to repetitive. You don't need to read Euler's work in order to indirectly benefit from it. It's the same for a lot of pure math.

Applied math does not use much pure math. It hasn't for decades.

This is not even close. Pure math is often complete clueless about how it applies. For example Developing algorithms to compute properties used in the most basic pure math is a massive pursuit.

As an applied mathematician, I'd say this depends on the field. Pure maths more often than not comes with assumptions that make the theory more elegant, but by doing so they also drastically limit its application to real life scenarios.

It is not true for Physics either.

Or computer science, at least when the code is also published.

(That said, performance measurements in most papers are complete junk. Most code written by academics is missing trivial, obvious optimisations which could completely change any comparative benchmarking results.)

I'm not sure that the average CS paper really tested the algorithm they thought they were testing. That comes from a lot of commercial projects where people weren't using the algorithm they thought they were using (even simple algorithms) and a few cases of working with code from CS researchers.

There was that time I got a C program from a famous ML researcher, tried to run it and it segfaulted, I pointed gdb at it and realized it was crashing before it got into main(). Understanding why was an educational experience (don't allocate a 8 GB array you never use on a 32-bit machine... It worked for the researcher because he had a 64-bit machine) but it did not increase my faith in the code quality of academic projects.

Some CS researchers are brilliant programmers but the fact is that ordinary devs make a living writing code and CS researchers make a living writing papers.

Allocating a big struct for everything and not using pointers is one of possible techniques of dealing with data in C programs. Anyway having a requirement of 8G RAM is not a good example of something "can not be replicated".

I know all about alternate ways to deal with memory, if I was just programming purely for fun I'd write assembly and not waste time moving the stack pointer around and other meaningless activities associated with calling conventions.

It's a sign of poor quality code.

He probably was using the array for something at one point, quit using it, and never bothered to delete it.

And of course it is C so there are no unit tests despite it being the kind of code that is eminently unit testable (if it wasn't written in C, not like you couldn't write unit tests for C but who does?)

> Most code written by academics is missing trivial, obvious optimisations

I agree with this, but I would pose it the opposite way. Academics shouldn't be prematurely optimizing their code: they don't have the resources for it, nor will their results be representative across all hardware. I routinely see (and publish) papers with novel cryptographic protocols where the reviewers will say things like "compare the running time of your algorithm to a state-of-the-art production system that Apple spent $50m developing and optimizing for their silicon, using hand-tuned assembly." We're not going to win in that benchmark, and we don't have the resources to even compete. Moreover, the resulting code wouldn't make for good reference code. But that doesn't mean we shouldn't publish our work, so that e.g., Apple (or other resources) can begin the process of optimizing it for production.

> Academics shouldn't be prematurely optimizing their code

While I agree with that in principle, the question of whether or not a given algorithm is fast enough to be practically usable is very useful information for the reader of your paper. Do I need to read your paper, understand it, implement it myself, and only then figure out that its 3 orders of magnitude slower than existing algorithms? I know its a lot of work, but as the reader I don't want to do that work either.

I've spent the last few years working on a novel collaborative editing algorithm for text editing. If the algorithm was orders of magnitude slower than existing CRDTs, it would never get adopted. And I want my work to be used. I want to use it, but again - only if its fast enough to be practically usable. I ended up writing a very highly optimised implementation of the algorithm - which took a massive amount of time - to answer that question. For the paper, I spent another couple of days writing a much slower reference implementation that people can read to understand how it works. (The optimised version is thousands of lines, and the reference code is a few hundred lines).

I understand how much work and expertise is needed to do things like this, but I still want someone to do this work. Even if its "We wrote a straightforward implementation of <competing algorithm> and <our algorithm> in plain C code. They perform at roughly similar levels of performance <see chart>. We suspect highly optimised versions of both approaches would also perform at roughly similar levels of performance."

Who's going to replicate an experiment that takes a $5 billion particle accelerator?

For that matter, who's going to replicate a condensed matter experiment for which a grad student took three years to build the apparatus at a cost of $250,000?

Peer review is an absolute joke. In principle researchers could publish their data sets and reviewers could see if they could replicate the analysis but they don't do it because they know if they tried they'd get different answers, see

https://www.nature.com/articles/d41586-023-03177-1

> Who's going to replicate an experiment that takes a $5 billion particle accelerator?

In the case of the LHC, which I assume is what you're talking about, this is why we have 4 pretty much independent detectors around the LHC looking at different but at least partially overlapping stuff. The Higgs boson was detected by the ATLAS detector, but wasn't considered to be "discovered" until it had also been detected at the CMS detector.

> Peer review is an absolute joke

That strongly depends on the field and the journal in question.

If they flew out the peer reviewer to interview the researcher and take a look at their apparatus or if the peer reviewer loaded up a Jupyter notebook and rechecked the analysis there might be some value in peer review. If peer review is just somebody looking at the paper superficially than it’s of very limited worth.

When I wrote my first paper somebody else had done a very similar experiment at another university and we wound method up peer reviewing each other’s papers. Turned out we were using the same bogus data analysis that was being used in 1000s of other papers. I was vaguely aware of this when I wrote the paper but did not feel psychologically safe at all in an environment where the American Physical Society claimed a PhD had a 3% chance of having a permanent career in the field.

Someone who was later to become a titan in the field was at our lab and was crying some nights because he had no idea where he was going to get his next postdoc wrote a paper about this problem more than a decade later after he had gotten tenure in a statistics journal that physicists probably won’t even read.

That sounds (as I said before) quite specific to the field you found yourself in. That isn't how things work everywhere.

For a relatively recent example - in my field of quantum information someone found in the course of reviewing some newer work that there was a mistake in one of the big foundational papers that this newer work was based on. They shared it with some of their colleagues and together they did some work finding out exactly where the error was and what the impact is on work that is based on it.

The response from the community was they got a talk slot at a pretty prestigious conference (QIP 2023, you can watch the video here: https://www.youtube.com/watch?v=2Xyodvh6DSY) and the work they did has been published here https://quantum-journal.org/papers/q-2023-09-07-1103/ and in a followup paper here https://www.nature.com/articles/s41567-023-02289-9.

My point is essentially that it isn't a big shock that this error wasn't found by peer review when the foundational paper was published in 2008. Even when it is working as well as possible peer review can only detect most errors, not all errors. However when your field is functioning "normally" and everyone does their jobs properly this doesn't matter because the errors that get past peer review get detected by other people building on the work later. On the other hand if your field is not functioning properly and people are knowingly doing stuff that is wrong then peer review can't help you. If the status of the field is so broken that people who know stuff is wrong are still pushing for it to be published then you aren't in a place where peer review can save you.

My opinion is you are incorrectly putting the blame on the (highly imperfect) process of peer review here where it doesn't really belong. Peer review as we perform it today sucks for many reasons, but it isn't to blame for the problems you saw.

It's not to blame for these problems but it fails to counter them.

In medicine see

https://www.cochranelibrary.com/

where there is a systematic process to search the literature and select papers with usable results. I've never seen more than 50% of papers get accepted and the other day I saw one where they found 80 invalid papers and 2 valid papers. It seems to me that peer review isn't accomplishing very much if it is passing through papers that can't be built on. For that matter, why is this work even being funded?

Yeah I wouldn't expect it to counter such problems. Peer review is by its nature review by other people in the same field. If you have a field where "bogus" methods are being used in thousands of papers then you aren't going to be able to get things reviewed properly.

This isn't an indictment of peer review as it is practised in every field.

I’ll say a good field is an island in an ocean of bad fields.

Look at hep-th. If in 100 years people conclude there is no evidence for supersymmetry and extra dimensions something like 90% of that field will be “not even wrong”. There is so much of the drunk looking for his lost keys under a lamp. I mean, there is a huge industry in 2-d gravity where it is possible to calculate things but gravity doesn’t attract in 2-d so the line between that and Newton’s apple is unclear. I’ve talked to astro-ph refugees who think that current theories about accretion disks are “not even wrong”.

Then there are all the folks who tell me firewalls make no sense because there is no such thing as the event horizon (locally) but from the viewpoint of people who fall in it is a tiny difference getting killed at the apparent horizon meanwhile the nonsensical “black hole information paradox” (sorry, no unitarity = no quantum mechanics) still gets breathless and insufferable articles written about it just because somebody famous for their disability made a bet. If young people were able to get established in the field you wouldn’t see people so fixated on bad ideas.

And the same was done for LIGO and the discovery of gravitational waves. 2 observatories, one in Hanford and one in Livingstone.

Yeah if one tried to find rot in academics, I think applied particle physics is one of the furthest from it. The scientists are super allergic to unfalsifiable theories that they start to doubt themselves when prediction is off by nanometers.

>>> For that matter, who's going to replicate a condensed matter experiment for which a grad student took three years to build the apparatus at a cost of $250,000?

That was my thesis. To make it worse, the laser I used became unavailable as I was finishing up. My work was adopted by 2 labs but required extensive adaptation.

> For that matter, who's going to replicate a condensed matter experiment for which a grad student took three years to build the apparatus at a cost of $250,000?

If the results are interesting to anyone and there is more science to do, people will eventually replicate (or fail to do so, like with Majoranas). If not, then why would we care about replication?

Did you write many papers in experimental particle physics? I did, and in my experience, peer review was not a joke. And results from different experiments really are used as validation. Citations are not just used as a metric, but helps to leverage prior work.

Sorry, what?

It's extremely true in physics. I've published in (good) physics journals.

Computational papers: Authors leave out details because they don't want to give up the goose that lays the golden eggs. They broadly describe the technique, ("solve this PDE computationally"), and anyone who wants to replicate has to craft his/her own solution. If they fail to replicate, the authors just say "your simulation was poor and probably numerically unstable".

This was the norm.

Experimental papers: Same thing. Experimentalists are inventors. They don't buy off the shelf equipment - they build their own. They don't provide details of how they built it in their papers.

At conferences, PIs would openly discuss whether they "believed" a highly cited paper. When I was a new grad student, this troubled me. Why should belief play a role? Just replicate it! A year later: "Oh, of course these studies can't be replicated."

It's one reason I left academia. I was doing theoretical/computational physics, but the whole game reminded me of English literature: The value of your work was always seen with subjective eyes. Your paper could be rejected simply because the reviewer doesn't "believe" it.

> Why should belief play a role? Just replicate it!

If you are talking about theory or numerical simulation papers, just replicating the calculations in the paper are not enough to believe the conclusions. There are many assumptions that go into any calculations, in terms of how much of spherical cow you treat reality as. Changing those assumptions will change the conclusions. The assumptions are not always clearly stated, and even the authors might not realize all their assumptions. It can sometimes take many years or decades for people in the field to figure out how a paper was wrong.

So yes, experts do have to depend on their subject intuition to determine how much they believe some surprising result. Not to mention, if you go through historical papers in your field you will inevitably come across decades long fights where one group is coming to one conclusion in their paper and other is coming to another, before the situation is resolved one way or the other.

Except it's the same problem for experimental papers. My thesis was based on a highly cited experimental study. My role was to "support" that paper with numerical simulations. And yet, at conferences, people would openly discuss if the original, highly cited paper, was fact or fiction.

The guy built/fabricated a device, and reported interesting measurements. Want to know if it's true? Build it yourself! Except when you do and get differing results, there will be quibbling over whether you built it properly. There are always details left out of the original paper that can be appealed to. The original author is king of the field, and people will simply ignore your contrarian findings.

For theoretical/numerical results, it is for sure valid to criticize assumptions. But in the longer time frame, the question shouldn't be "Are these assumptions valid?" but "Is there experimental evidence to support these calculations?" With a lot of the papers published (including mine), I can assure you that no one will ever be able to construct an experiment to (in)validate my results. And that was true for probably over half of the theoretical/numerical results in my field. We were all publishing things that most of us believed could never be connected meaningfully to the physical world. We're not talking string theory or high energy stuff - more like material properties.

The goal was to publish, and convince others. Not to understand reality. Hence, more like literature.

I dunno. When I was in grad school I regularly would find papers in Annals of Mathematical Physics that would go on for 50 pages and have critical errors on page 17 and 45. I did a calculation like that which I know what right at the equation-by-equation level because I made unit tests for everything, but the whole thing was still useless (published in my thesis but not Annals) because the coordinate system we were using converted a periodic orbit into something like a spiral so the we couldn't compute a topological factor because there was no topological invariant...

I've published in some well-respected, but not highest-tier, journals (think impact factor ~ 5-7), where this was absolutely the case. You could very often identify the reviewer by who the corresponding author of the twenty papers they asked you to cite was. Perhaps it's field dependent, my field (chemistry) was not always as, shall we say, ethical as some of the other fields.

… I thought people in physics were pretty sportsmanlike but my understanding was that biologists would stab you in the back for the slightest advantage. I would complain about how hard it was to make it in physics but in biology it seemed almost impossible to get established since young researchers just couldn’t get grants period.

I had on old physics professor who said that by the second page of any math/physics/equation-heavy paper, assume all signs are randomized.

I've discovered in the intervening years since graduation that he was wrong, you should start assuming by the bottom of the first page.

I told my thesis advisor that!

I don't know about that. IIRC the creation of HoTT was because someone was notified that one of his important papers from earlier in his career had a non-trivial flaw in it.

To be sure, reproduction is a bit easier in mathematics because you normally don't need exotic equipment, just time. However, that doesn't mean that people are necessarily taking the time to double check everything with sufficient rigor.

In the article below, they claimed one study said a third of the papers had a false theorem.

https://news.ycombinator.com/item?id=17430577

A number of papers on formal verification reported that restructuring their specifications for proof revealed problems they missed in the specification. Other works found through empirical testing that the specs didn’t match reality. Many theories in physics changed that way.

So, we should assume they’re all false by default until the specs are reviewed, the proofs are checked, and there’s empirical validation. If theoretical, its component theories must be built on observed truths instead of imaginary constructs. Then, we might trust the paper (i.e. mathematical claims).

Is replication a big problem in computer science?

If anything, I'd think replication has gotten easier with the advent of open source and the expectation that top papers will include git repos which make it easy to reproduce. The advent of docker, locked-version-dependencies etc have all contributed as well. 10-20 years ago, we didn't have this - but I also didn't see * widespread * problems even then: if someone published a result saying that technique <x> led to <y> performance gain, I thought it usually did?

(that said, computer science research doesn't have the same money on the line as medicine and the "perish" part of the publish-or-perish equation isn't the same when the fallback for failed academics is still six figure salaries at tech companies)

What are the odds that in 2044 we will still be able to run the code published with CS papers today? If it's pure Java SE or ANSI C with no dependencies then it will probably still work, but anything more complex tends to decay pretty rapidly.

Most CS papers are theoretical, and this is another reason why HN is wrong about requiring papers to publish code.

> The truth is that almost all papers published across almost all fields cannot be replicated, full stop.

Isn't it ironic that we can't replicate your assessment?...

I personally have no idea how much papers can be replicated or not, but replicating an experiment would not guarantee that the idea/conclusion in the paper is sound (ex: paper extracting some data, observing a correlation and concluding it is an implication).

I do agree that it should be made easier to replicate for the fields where this is possible, but there are many other problems.

this is overstating the case. Throwing out all publications is ridiculous. Yes, many fail to replicate, but not nearly all. A larger concern may be that many fail to generalize (e.g. mouse cancer cures fail to work in humans).

It's true that it's hard to ignore all of the citations a reviewer suggests, but that's part of the role of an editor, and the response to reviewers (while being a frustrating exercise) can make it clear what is and isn't an appropriate citation, which serves as communication to the editor. Yes, this process is frustratingly low bandwidth

How do you know? It's rare that anyone even tries to replicate scientific results outside of clinical medicine

Exactly. What kind of editors does your field have? In my field, that kind of oversight would have been called out on twitter immediately.

> The truth is that almost all papers published across almost all fields cannot be replicated, full stop.

And yet we still have advances in technology. I wonder how this academic nihilism matches reality?

This is where I got cognitive dissonance. Can both actually be true at once? If not, which one is wrong?

Reply on news.ycombinator.com