Hacker News Reader

> My pet idea for making science less wrong is to maintain a citation graph of all papers. Then when a problem is found with any paper, every other one downstream from it is automatically flagged as at risk of being wrong. Now all those authors (or others) have to go back and re-evaluate how that citation was used and how it affects their result. Once they decide it's OK, they update their paper and remove the flag. If it's not OK then their paper is retracted or flagged as wrong or simply keeps its at-risk flag if it's not important enough for anybody to bother re-checking it.

This is an interesting idea, but I don't know what real-world problem it would actually solve. As someone who reads a lot of scientific papers, I can't think of a single one whose correctness is actually dependent on the correctness of the results of a paper it cites, as though the paper in question is "buggy" because it "calls another paper as a subroutine" and that subroutine is buggy. Could you give an example or two of this?

I could see this making sense in mathematics since mathematical proofs really do "call other papers as subroutines" (theorems), but not so much in science.

>"As someone who reads a lot of scientific papers, I can't think of a single one whose correctness is actually dependent on the correctness of the results of a paper it cites"

What field are you reading where each paper is independent of all the others? As the most obvious example, in biomed pretty much every paper says something like "as done previously[ref x, y]" in the methods section. It is an extremely annoying, but very common, practice (usually refs x and y contain somewhat different methods descriptions and/or also contain the "as done previously" phrase). There are many other ways the papers depend on each other too ("we used these cells because they don't express this protein[ref z]", etc)

I more or less exclusively read CS papers, which sometimes contain similar statements. I don't see how in your example, "ref x" being invalid would make the paper citing it invalid. Can you give an actual example of a paper that does this that could be invalidated in this way?

Basically, to me, an empirical result arrived at using the scientific method is, well, an empirical result arrived at using the scientific method. And it seems to me that the only thing that would invalidate such a result is if it were found that the researchers did not follow the protocol they described in their paper. The fact that the researchers might have merely cited another paper, or used the method described in another paper, in which the researchers were found to have "cheated" in this way, doesn't seem to invalidate the paper in question. It's possible that the researchers in question could have happened to also cheat, but I don't see any reason to assume it's any more likely because it happened with a paper they cited (unless of course they're the same researchers building on their own previous work).

For the methods citation example, the citation chain (is supposed to) go all the way back to the paper devoted to originally describing the method. In that paper there (is supposed to be) a bunch of checks (compare to other methods, etc) that the method is working as claimed. These checks will not be repeated every time.

    > > whose correctness is actually
    > > dependent on the correctness of the
    > > results of *a* paper it cites

    > where each paper is independent of
    > *all* the others

These are not describing the same thing

If I understand, the author is suggesting something like pagerank for axiomatic correctness where a paper's score can be interpreted per paper (in terms of some binary promise of "correctness") even if the score is computed by the weighted correctness of the papers it represents.

Really, the information should be abstracted away from the paper, a la freebase or wikidata, and there should be strict rules (e.g. the protocols stackoverflow enforces) such that no cycles can occur, such that propositions can be resolved to axioms, and such that it is practical to compute scores in a finite period of time with reasonable resource constraints...

See also gitxiv.org (at attempt at literate research for computer science and physics papers that can be paired with code). It doesn't address the problem described, but if statements of a paper are codified (e.g. curry howard isomorphism) in a strict way, it makes it easier for "linked statements" to have a quantifiable value.

I think they are, unless you are selectively citing (irrelevant) papers to avoid depending on any...

I mainly read papers of second language acquisition which is an empirical science that performs language learning experiments. Often the "inspiration" for the research in newer papers stems from the results of earlier ones. The empirical results of the newer ones are, of course, not directly invalidated if there was a problem with an earlier one.

However, some of the papers tend to discuss the results in the light of the current empirical results AND the earlier empirical results ("from this result, we now see that A → B, and from the earlier results we already know that B → C, so there's some evidence that A → C"); this of course biases the discussion and may lead to wrong conclusions.

By the way, the largest problems I see in the papers are not directly "wrong" results, but like the "Chinese whispers" or "broken telephone" game. The paper A makes some assumptions and uses certain test protocol, measurements etc, and then paper B refers the result but misrepresents it slightly, which may cause overgeneralisation. A concrete example: paper A tests the learning of some phoneme with a training period and before-after test and publishes the result. Then the paper B later references the paper A mentioning that A has shown that the learners "learn" the phoneme. However, represented this way – "unqualified" – this is an overgeneralisation: the students haven't actually shown to have learned the phoneme in any other way than using the test. Indeed – other studies have shown that using certain kind of tests are generally very poorly correlated with what people usually mean by "having learnt" something – being able to use it in spontaneous communication.

What would be your plan for papers that cite another one because they are doing a literature review or something akin to "Paper X is wrong"?

There's also a trickier problem in your simple binary flag: papers can have incorrect proofs of correct statements.

I'd love to see a Semantic Web approach where you don't cite papers but results in papers. So while a result in a paper might be wrong, you don't have to throw out all the results in all the papers citing this paper.

@averagewall, check out openknowledgemaps.org

Also, citeseerx and crossref are your friends.

Pubmed central similarly has the ability to return much of this citation data programatically (for certain scientific subdomains)

If you're passionate about prototyping such a service, let me know and we can get you an invite to the Archive Labs slack (we're a non-profit, volunteer-run community which incubates open access and public good projects). Literate, reproducible research and knowledge + citation graphs are two of our emphases.

Good luck

P.S. One way you might describe this project is "an axiomatic Euclid's Elements for all scholarly knowledge".

You might also like metacademy.org (one of my favourite projects -- it's a package manager for knowledge built on DAGs)

You could prototype this with the data in crossref at the moment. They don't have full citation coverage (basically nobody does, it's a bugger of a problem), but there should be enough there to try out some ideas with real world data (and free data with an API).

> We would first need to solve the versioning problem where there's no way to update a paper when mistakes are discovered or simply to improve it.

You can certainly update what a DOI points to.

Surely this exists.