Propagation of mistakes in papers

That experiment is also used to teach about selective data exclusion (potential scientific fraud) as well as a resistance to challenging an already published value (future experiments searching to verify Millikan's value rather than show it was incorrect or off) in a lot of experimental classes.

https://en.wikipedia.org/wiki/Oil_drop_experiment

Is that part of the genesis for these conversations about how perhaps the physical constants of the universe are slowly changing over time? If you look at the 'right' experiments, the speed of light slowly crept up over time too, IIRC. When the movement is all in one direction it's easy to speculate that maybe that's because the target keeps moving.

Another example I recently came across was the early measurements of the AU using radar. The first two experiments tried to bounce radar off of Venus had very noisy data, but they seemed to have a detection that implied a distance that was pretty close to the earlier measurements that had been done using parallax. But after the equipment was upgraded, the detections went away and it turned out that they had just been noise. Later on an even more powerful radar system was able to successfully bounce a radar signal off of Venus and it turned out that the AU was quite a bit different from its earlier value.

This is somewhat a criticism of how contemporary citations work though.

Primitive science (or even pre-publishing science) doesn't get cited because humanity figured it out before our current system was in place.

It may sound silly, but no one feels the need to cite Eratosthenes when implying the world is round.

But many people do feel the need to cite the colorimetric determination for phosphorus (an SCI top 100 paper) even though it was published 100 years ago and is generally considered “base-level science.”

It is certainly an interesting paper to read, but I’m not sure I need every scientist to read it in order to believe they know how to do colorimetric analysis.

I'd be curious to know whether the percentage of incorrect citations varies over time. I would guess more recent authors would be more likely to search by title in Google Scholar or SciHub (or use the DOI link, if available) rather than actually use the volume and page number, which could result in more authors who did read the article nonetheless getting the volume number wrong.

Or maybe that's because somebody published a bibtex entry for that paper that got that volume number wrong and those people just copied and pasted the entry without reading.

There's a semi-famous line of research by Simkin which uses citation copying errors as 'radioactive tracers' to estimate the rate of copying & nonreading, under the logic that (in a pre-digital age), you could not possibly have repeated the '113' error if you got an ILL copy or physically consulted volume '13' (if only because you would be pissed at wasting your time either checking volume 113 first or verifying there's no such thing as volume 113):

https://www.gwern.net/Leprechauns#citogenesis-how-often-do-r...

Your 10% isn't far off from the 10-30% estimates people get, so not bad.

Solid point. The paper is a delta-mapper: it provides a p --> ∆p prior to posterior-change. However, it does not tell you anything about p or p'=p+∆p itself. To get true value of p^n, we sum over all ∆p in some way (affected by the path we take through papers addressing).

You're modifying the thing so that future ∆p^{i+k} are added to the delta-mapper so that ∆p is appropriately modified accounting for that ∆p^{i+k}. It's like path-compression in a union-find structure.

It is interesting as a helpful approach but does suffer from the pingback spam problem, right? And I have a slightly sneaking suspicion that it is not an accidental oversight in science that leads to these problems.

> Namely, paper references always reach back in time. Papers don't reference papers that were written after they were written.

Actually, the opposite happens quite regularly: the author X got a pre-print of a paper by author Y. From Y, he knows that the paper will be published in the future in an upcoming volume of a scientific magazine. So, X already writes this future reference into his paper.

Somewhere near 100% of my shipped bugs have been peer reviewed so that makes a lot of sense to me.

> To be clear, I think getting through peer review is evidence that a paper is good

I think this depends on how you define good. I'm sure there's some variation across fields, but peer review generally seeks to establish that what is presented in the paper is plausible, logically consistent, well-presented, meaningful, and novel. That list is non-exhaustive, but correct is very hard to establish in a peer review process. In my experience, it would be rare for a reviewer to repeat calculations in a paper unless something seems fairly obviously off.

As a computer scientist, it would be even more rare for a peer reviewer to examine the code written for a paper (if it is available) to check for bugs. Point being, there are a lot of reasons a paper that appears good may be completely incorrect. Although this is typically for reasons that I as a casual reader would be even less likely to distinguish than a reviewer who is particularly knowledgeable about that particular field.

I've got a close relative who reviews papers all the time in their field (not CS). Based on that my take is that if a paper passes peer review it is a good indicator there's nothing egregiously wrong with the stuff that's written.

I just completed a paper review as a reviewer. After I think 4 rounds, the author finally ran the calculation I had asked for in the initial review and admitted I was right. We got there in the end, but I had to sit on my hands.

Peer review can help improve a paper (and it has improved some of mine); however, contrary to some popular notions, it doesn't lend "truth" to a paper.

Peer reviewers are not monitoring how experiments were conducted, they only have access to a data set that is by necessity already highly selected from all the work that went into producing the final manuscript. The authors thus bear ultimate responsibility.

When considering published work close to mine, I use my own judgement of the work, regardless of peer review or which journal it is published in (for example it may be in a PhD thesis). For work where I am not so familiar with the methodologies, I prefer to wait for independent verification/replication (direct or indirect) from a different research group, which ideally used different methods.

How do you CI/CD assertions in papers using animal models in experiments that take months to years?

How would that work? You can't automate testing of papers. However flawed the process is, this is what peer review is intended to do.

A billion dollars worth of AI and formal methods research