Why is there no semantic ontology of sentiment in academic citations?

> Besides, in many fields, citations are provided just for completeness. "Previous studies have dealt with -TOPIC- (-CITATION_1-, -CITATION_2-, ...)" is quite a common construct in the natural sciences, for example. Exactly what information can be gleaned from this is uncertain to me, beyond "-CITATION_x- CITES -CITATION_1-", "-CITATION_x- CITES -CITATION_2-" and so forth. And this citation linkage is already available.

I published one paper before leaving academia. It was published in a top journal in the field and has about a dozen citations now (I get a notification email each time a new citation pops up). Every single one of them is this kind of citation-in-passing. Not a single one actually engaged with the content of the paper.

I have a couple other half-finished papers that I was working on before I quit, and sometimes I think about going back and polishing them up for publication, but given the reception my previous paper received it's hard to justify the time. It's not like I need papers for tenure now.

There's been something kind of like that in the field of law in the US for around 150 years.

When you were doing legal research on a topic and found a case or statute that you wanted to use you could look that up in a multi-volume book set called "Shepard's Citations". You would look up the case or statute in Shepard's (which people called "Shepardizing") and Shepard's would list almost all the other cases and statutes that cited the one you were interested in, and for each would tell you if you if the later case or statute followed yours or overruled it.

Shepard's is still around but now online, although the print edition is still around.

Shepard's deals with the problem of finding out what happened to a case or statute. Another problem in legal research which was also solved with multi-volume sets of books is finding cases and statutes of interest in the first place. There were books that organized all of the law into a large topic outline, assigning codes to each topic, and books that published cases and statutes with notes summarizing what topics were covered and what the case or statute said about them, and books that indexed those books by topic.

You could go into a law library to research a topic, use those indexes to find relevant cases, use the summaries to see if it is likely a case you need to look deeper into, and Shepardize it to figure out if it is still good.

It was essentially a database system, implemented entirely in books.

That said, I have hope we can solve this problem outside the academic incentive system. Here's a relevant prototype developed by my research lab: https://braid.news/about

While the parent is right in stressing the importance of the review phase, which is a filter to decide whether we will ever see a candidate paper or not, it is indeed the case, as the OP posits, that there is merit in extracting the context in which a citation is used, which has been termed "citation polarity".

For example, 'Chomsky (1969) was entirely wrong when he said that “it must be recognized that the notion of 'probability of a sentence' is an entirely useless one, under any known interpretation of this term".'

There are Natural Language Processing methods and tools to extract the citation polarity. Researchers like S. Teufel have used citation polarity and other methods to analyze publications ("rhetorical zoning") and to create maps of the published literature: https://www.cl.cam.ac.uk/~sht25/az.html

A scientific literature search engine (Semantic Scholar, CiteSeerX etc. - sadly Microsoft Academic has recently been discontinued) can benefit from such knowledge; for example, PageRank can be modified so as to incorporate citation polarity in the random walker model that underlies it. Think of it as adding a "dislike" button to a system that already offers a graph with "like" button relations.

I disagree that that is the sole (or even most important) point of a citation. Most of the time you can view a citation as part of the (extended, implicit) problem statement for the paper, e.g. here is what X did in this space, and we are different/better/build on that work by doing Y. If an important citation is missing, it shows you aren't familiar with what has been done before in relation to the problem you are trying to solve (bad). Of course reviewers and readers are free to disagree with your characterization of a cited paper in relation to your own.

Having said that, maybe if you publish in journals a lot this is more of an issue. In conferences the Program Committee is usually set in advance so there is less scope to target specific reviewers (other than those on the PC).

This is a very cynical and opportunistic way to do science. It may be the winning strategy to play the academic game and advance your career, though.

Citation rings are a huge problem, but couldn't the sentiment-analysis side be readily addressed? There's a difference between papers that just cite a paper without judgment (+0), papers whose results explicitly support another paper (+1) and papers that explicitly reject another paper (-1). Such an AI could also bypass "citation analysis" and instead ingest the entire scientific literature, and find refutations in papers that don't even cite each other.

In practice, 2 problems: 1) this requires sophisticated AI; and 2) many people will keep assuming that high citations = credible (i.e. assume that +0 and +1 are equivalent).

But this method will help researchers go beyond "it's highly-cited therefore I trust it"; it'll help students and rigorous scholars find how well-supported a paper is, despite its popularity. And it'll surface refutations that may be hard to otherwise find; for example when a scholar that's famous in a field, and unknown in another, decides to publish a paper in the latter field that may not be widely seen.

Ultimately (though this is far off: theoretically completely possible but highly technically difficult) we can also imagine AIs that ingest the whole literature, gains deep cross-domain knowledge, and are trained to detect poor methodologies, or automatically highlight new insights in one discipline that can enrich another discipline, or (ideally) make its own judgments about the merits of any finding based on the knowledge it acquired. After human review, it could systemically help "clean up" the scientific literature of bad methodologies and ideas, and free scientists to spend less time working on scientific dead ends.

That last paragraph won't be doable for a while, but the first 3 seem readily within reach of today's AI technologies (and seems to be what scite is doing).

>My advisor taught me that the point of a citation is not to give your opinion on which other papers are good or bad; it's to tell the editors which other scientists should be on your paper's review committee

As a professional consumer of academic papers paying commercial rates for decades I appreciate your candor profoundly. Nevertheless what you say is a statement at least of disingenuity and in my opinion prima facie admission of the existence of a defrauded system.

Absolutely. Peer review causes peer pressure.

Ofc, it depends heavily on how big one's 'niche' is, but...

Yes, using NLP (DL) techniques makes more sense than to require every author to annotate their work.

This seems to be very hard to pull off, though, given the way people regularly cite work in some scientific domains. For instance, unless specifically comparing to an approach, researchers in ML rarely endorse or disavow a specific approach (just my observation in graph learning).

The nature of citations are often much more ambiguous, though, and assigning sentiment will often be unhelpful.

Background citations are often of the flavor of: "These guys did almost exactly the same thing as us, but due to the minor detail of wanting to get our own work published, we think their work is inferior."

Or, "This cited work had a great idea, but due to rush to publish they left out a bunch of details and took a few short cuts on the implementation. We've fixed some of these holes, and called the original authors dolts to elevate the importance of our own incremental improvements."

Read the paper; ask someone else who read the paper :-)

Your best bet would be to find a review paper that cites Smith (2015) and see how they interpret it and how Smith (2015)'s conclusions jibe with the rest of the review. A few review series even use little icons to indicate "important" and "crucial" papers. Even there, you've got to be careful that you haven't fallen into some weird fringe-y subfield though.

Raw bibliometrics aren't totally useless--I'd bet most bad work is ignored, rather than refuted--but I'm really skeptical that anyone can automatically put a valance onto most citations.

Came here to mention CiTO. See also https://sparontologies.github.io/cito/current/cito.html. It’s been around a long time, from when the Semantic Web was the next big thing.

It's worse. Most academics don't even make their citations links, even though this is probably the most obvious use case ever for links. Instead, we get PDFs that really want to pretend they're paper journals, and citation formats that try to just infodump enough for you to go find the paper yourself.

Citations take a lot of time already as it is. Any extra metadata would take a ton of time and for what? Just so some data nerds that wouldn’t read the paper anyway can play with it?

The problem is that there's more than one view on ontology, and the dominant view changes sometimes.

Look at biology, where the knowledge is expected to be based on directly observable, highly repeatable phenomena like animals or plants. The tree of life that orders the species, families, etc up to the realms at root is constantly reviewed and reordered close to leaves, and had seen a few changes close to the root.

Ontology, being a lot less strict than disciplines studying more direct observation, would have much trouble producing a universally agreed-upon tree of notions, and a lot of energy would be spent around contested branches.

I suppose the only possible outcome is an ontological forest, or shrubs, like what we have now, which is only locally helpful.

> Imagine what it would do to literature review to draw from a 'tree of knowledge' of all the supporting texts of a work, or to see all the work that claims to derive particular meaning from some seminal work.

You can visualize all of these things using Scholia https://scholia.toolforge.org . Built on free and open citation data, no need for Google Scholar or anything proprietary.

That sounds interesting. Do you have a link? Or more details? Ta.

The thing you are missing here is that citations are often intentionally ambiguous as to whether they are neutral or negative.

Or, phrased differently, there is essentially no possible benefit for a researcher to say "this piece of related work is junk". Let the readers decide or infer instead.

How would you treat something like "They had the right idea, but their simulation was in 2d only, and we know now that this makes the simulation results garbage. They just happened to get lucky and got garbage that matches the real thing in 3d" with that?

This is probably something that varies by the field, but I don't think most citations can be meaningfully placed on a positive / neutral / negative axis. When I cite something, it usually falls into one or more of the following categories:

- background reading

- related work

- source for a specific claim / result

- credit for software / data / other resource used in this work.

I don't evaluate the work when I cite it. Instead, I try to provide useful information to the reader and to give credit when academic courtesy requires it.

>There aren’t meaningful disagreements in many disciplines as those who disagree and are smart are selected out of a field.

This trope needs to die. There are major competing theories in pretty much every academic discipline. The claim is so ignorant and baseless that I think less of this site every time I see it come up. It is the HN version of a shitpost.

We get it, you don't like academia. To be honest, there's a lot to dislike. But please, for the love of god, stop writing this low-effort nonsense on a site that is meant to have users who actually use their brains.

The legal example is actually maybe even more interesting than you may think. The big traditional databases have always employed human reviewers to identify cases that overrule or question other cases. Attempts to automate this process by e.g. Casetext have had somewhat limited success, despite the fact that this is a restricted context with formal language. It’s getting better, but whether it will ever be good enough is still an open question.

See https://en.m.wikipedia.org/wiki/Shepard%27s_Citations

(By the way, signals are not much used in opinion writing; that aspect of the Bluebook is geared more towards law reviews. Approval or disapproval of prior authority is in the opinion text itself, as this is a formal activity of the court. Notation as to whether a given cited case approves or disapproves another case would be in a parenthetical after the citation, e.g. “Smith v. Jones 123 Foo2d 456 (1999) (overruling Jones v. Smith 12 Foo2d 345 (1989)).”)

How do you control for bias in the post-doc? What exactly is "sentiment"? Is it a qualitative description, or something more analytical? Why does a future researcher care about the sentiment instead of the content itself? Or is this to better help guide other research directions?

Seems like a good project for an undergrad who is just getting started in research, actually.

We just call this "literature review", and everyone does it at times regardless of seniority level. But the aim is not a sentiment graph because it doesn't really matter whether author A said paper B was good or bad. What would we even use that information for? We instead want a comprehensive list of papers that are relevant to what we're trying to do right now, with a note on each saying why it is relevant.