Inciteful: Using Citations to Explore Academic Literature | Hacker News Reader

Inciteful: Using Citations to Explore Academic Literature

inciteful.xyz

144 pointsadamnemecek47 comments

Creator here! Wasn't expecting this to happen :) The site is definitely still in Beta so I appreciate any and all feedback. I just launched it a few days ago. It's been my COVID project and I finally got to the point where I felt comfortable having others use it.

The biggest hurdle was the speed of the graph creation. Basically taking a 250,000,000 paper/2,500,000 citaiton db and creating graphs that could be up to 200k papers and 3-4mm citations. For that I ended up learning/using Rust (which was a great experience).

The plan is to keep it totally free and hopefully get some institutional support once I get a better handle on demand and costs.

Ask me anything!

EDIT: As you are going through the site, be sure to use the purple "+" buttons to create your own graphs centered on the topic of your choice. That combined with the in-graph keyword filters are probably the most powerful ways to quickly zero in on the most relevant literature.

Which crate are you using for graph manipulation?

I go through a lot of the details in my post on the Rust subreddit: https://www.reddit.com/r/rust/comments/kfiaqn/just_wanted_to...

But long story short, I end up doing most of the graph analysis by passing in the citations, using PyO3, to graph-tool in python then returning the data I need about each paper. I am planning on moving that over to Rust. But not being an academic I wanted to get feedback on the quality of the results before making it difficult to quickly test different types of algorithms.

Are you planning on open sourcing parts of it?

Eventually I'd like to move the site to open source, but right now the repo isn't in a place where I can do that. As for specific parts, it's pretty purpose built and this is my first Rust project and so I'm not sure which parts would be helpful to the community. And I doubt they would meet the communities standards just yet :)

It’s better to open source sooner rather than later even if it’s not in a place you’d want it to be. Like some of the work you have to do might be done by the community.

Very nice work. I especially liked the ability to build up a collection of papers, that the response time was good, and that the SQL could be edited directly.

Do you have any plans to add a graphical visualization of top/central papers?

That is the most requested feature and something I'm working on. It's a fun (and hard) design/data problem. Which of the 5k-150k papers do you show in the graph? And then how do you render them in a way that is both visually appealing but also conveys the most import information?

Displaying graph data in a user-friendly way is indeed a challenge. The nicest user interface I've seen is on metacademy "explore" views:

https://metacademy.org/graphs/concepts/bayesian_logistic_reg...

and src code for the graph view is here: https://github.com/metacademy/metacademy-application/blob/ma...

they do some clever hiding of edges so graph is not overwhelming, but still only O(100) nodes.... for O(100k) nodes you'll need to do some selection for sure ;)

Just curious, what source are you using for the citation graph? I seem to remember looking for an API to something like ACM digital library at one point and not really finding what I wanted, but maybe I just didn’t know how to look.

I love this idea btw, I’m going to use it to find some holiday reading!

I'm a big fan of trying new search engines for academic research; I hopefully am more likely to break out of whatever search bubble I'm unaware I'm in.

This one in particular had some very nice features, some of which are present in Semantic Scholar (my current favourite) but some which are certainly not.

Recommending papers based on citation graphs is a good way to very quickly get up to speed with fields I'm not to familiar with, but I'm always wary that I'll end up back in the feedback loop of very few popular papers rising to the top while perfectly good papers go unseen because they weren't well cited in the year they were written.

So I'll certainly keep an eye on this and give it a try, but I'm certainly still in the market for a "serendipity" slider on such recommendation engines.

If you have the chance, try the following:

1. Find a paper you like in a field you want to learn.

2. Use the keyword filters to filter down to papers that match your criteria.

3. Add a bunch of the interesting ones to a new graph using the purple "+" buttons.

4. On the "new" graph page, check out the similar papers section. If any of them are interesting, add those to the graph.

5. Repeat until you don't find anything else that is interesting.

The similar papers section uses a link prediction algorithm that basically says, if two papers cite a bunch of the same papers, rank them higher BUT if the paper they cite, is cited a bunch of times, don't give that connection much weight. The net effect of this is that it doesn't really matter if the paper was highly cited, only that it cites the same niche of papers as the ones you just chose. Also, because of the temporal nature of academic literature, the papers it brings up tend to be the newer and harder to discover papers.

The results are pretty great and it's as close to the "serendipity" slider that you'll get right now.

EDIT: Formatting

I'll give it a shot!

8 mentions of the 'graph' on the page. Zero renderings of the graph :(

It is a nice user interface and the reference material is useful and well presented. But when you get down to it, it's linear lists about the characteristics of a semantic graph. As I've said many times, this is like describing a tree with a tabular catalog of its leaves. Graph navigation needs a graphical representation, because things like branchiness (node out-degree) and other factors are more easily shown than described.

The basic problem with graph representation/ navigation/ traversal is that there are many valid ways of looking at the graph and it's hard to render them all. Maybe try using a gutter to allow users to temporarily pin certain graph characteristics and render accordingly. In this context, sometimes I might be interested in the latest research that cites a paper, other times I might be looking to see who picked it up first, or to apply some sort of windowing function to a large spectrum of citations.

But I want to see the graph, even if I am looking for a particular leaf.

Good feedback. Visualizations are definitely the most requested feature and something I'm working on. It's a hard problem to distill so many papers (5-150k) down into a usable graph. I've made big ones that look amazing but which are not that useful.

I like the idea of interactive filters to allow the person to explore it visually. I'm hoping to have something people can play with in the next few weeks. I hadn't really expected the site to take off the way it has and so it's not really feature complete yet :)

It's outside of my technical wheelhouse, and most of what I know about graph theory is from trying to get a better handle on my domain problems, and as I get older I'm increasingly reluctant to get stalled on toolmaking.

But what you're discussing is a frequent problem with graph visualization - it's very easy to end up with 'hairball charts' that may be meaningful to the person who generated them but only because of familiarity with all the steps it took to get there, and the more inclusive the graph, the more time eaten by clustering algorithms, the more clusters produced, and the more of a cluster...well you get the idea.

As you're at an early stage of development, perhaps this technique, which trades away completeness for clarity and is relatively novel, might let you leapfrog some of those problems: https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8896846

This looks super interesting! I'll take a look. Thanks.

I've no idea about Rust, but in Python land you might want to check out Graph Tool and similar tools. As you note, layout of large graphs is not trivial.

First impression is positive; relevant results and a reasonably straight-forward UI. I appreciate the warning about the slow-loading graph (which did load after a minute or two).

Two things that might be tweaked:

* The search didn't behave in an ergonomic way: I typed a query ("graph neural networks") and great relevant stuff came up immediately in the dropdown. When I hit enter, however, I got an error that read "Invalid search: Check your spelling, enter a DOI, or another paper identifier or." I would have expected my action to take me to a search results page that listed what I saw in the dropdown (which I regard as a preview of the top hits) so that I could peruse the selection carefully.

* I wanted to load a paper to take a look at it and it took me a while to realize that I could click the "Yes" above "Open Access" to download it. Since one of the big use cases for a site like this is the eventual consumption of these papers, I suggest making a "read/download paper" call to action more explicit.

Thanks for the feedback. #1 might be some sort of bug. #2 Good call!

Here’s the original announcement https://www.reddit.com/r/rust/comments/kfiaqn/just_wanted_to...

Hi weishuhn, good work, I'll be using this.

Can you tell me how you're managing copyright? In my understanding, you're effectively "republishing" abstracts, possibly altering or remixing them too. Presumably the site isn't a commercial concern yet, so there's probably little issue. If the site starts to generate revenue, even if it's just to cover costs, or if it becomes a registered business, does this change the copyright situation for you?

I see that you're using several sources of open data, so perhaps all the data you're using is free from copyright, or has highly permissive copyright e.g. CC0.

For those wondering about open-access, in my understanding that's about _reading_ papers. Putting papers on a website might make the website provider subject to copyright. This applies even to abstracts.

As a minimum, it might be necessary to have a button on your site, weishuhn, to report any paper that has restrictive copyright. You've already noted one example of misclassification, and I know that Crossref and other sources provide their data with a big caveat that it might have inaccuracies. Responsibility cascades to you, unfortunately.

This inevitably leads to a trade-off between completeness of your data vs usefulness of your service. It's a problem I'm wrestling with too!

None of this is criticism, just thoughts from someone working in a similar area. As others have said, it's always good to see innovation like this in literature searching. Good work!

Is this any better than Connected Papers? https://www.connectedpapers.com/

Creator here. It's similar in that it uses citations to make paper recommendations. But different in that I give you access to the entire paper graph rather try to distill it down to just a few. You can even write your own queries by clicking on the "SQL" button at the bottom of each table. I kind of view it as a Connected Papers for power users.

Thanks for the response! I have enjoyed using Connected Papers so I will give Inciteful a closer look!

Relevant to my interests, thanks.

I played a bit with the site and liked it. The one killer feature I might pay a few dollars per year for is to create a profile, with a few graphs attached and get a daily or weekly email when a new paper is published that fits well into one of my graphs. Feel free to add a monthly "the most important old paper that you have not read yet (or not read in the last 5 years)" email too.

this seems very nice -- very polished and gives good results.

I would like it if the bibtex entries had meaningful cite keys as opposed to long numbers. as is, it would be pretty difficult to actually write a paper using these bibtex files.

I am planning on making the info in the BibTex files a bit more robust. Right now I'm just adding what I have readily available. But in general, if you are using Zotero or Mendely, the functionality to enrich the metadata on the entries does a good job filling in the missing info.

In case the creators of this site are reading, there are some grammatical issues with the front page copy:

“on it's head” should be “on its head”.

“not only with” should be “with not only”.

“analysis'” should not have an apostrophe and should possibly be “analyses”.

Much appreciated! I'll update it. I really hadn't anticipated much traffic. I was planning on doing an open beta but things have kind of taken on a life of their own (in a good way).

Cool idea, but I think I broke the search with quantum computing. No papers to download, no links, and no related pages. Just 404s

https://inciteful.xyz/p/186039733?&keywords=hello&maxDistanc...

You had "hello" as a keyword. I removed it and saw a bunch of papers pop up.

How does the engine determine open source?

I checked a paper, 10.1097/RHU.0000000000000563 for example; which I know for a fact is open-source marked as non open source.

Otherwise, very nice tool. Will be using this regularly. Awesome stuff.

For that I use the Unpaywall API:

https://api.unpaywall.org/v2/10.1097/RHU.0000000000000563?em...

According to them it's closed but I definitely see it as open here:

https://journals.lww.com/jclinrheum/fulltext/2017/09000/omeg...

I'll look to see if I can find another data source for OA papers to add more coverage.

How do you make it work with arXiv documents - I'm copying/pasting the ID and it's not finding it. Bug or feature? PS: very good idea, cheers.

The database is about a month out of date right now. I am going to be updating it soon. You should be able to either put in the url or do arxiv:XXXX.XXXXX

Curious about HN's rules on titles. How is the language relevant in any way to this particular post? That sure seems to be editorializing in an attempt to promote something other than the project itself (and obviously to get upvotes). Given the explicit rules against both editorializing and soliciting upvotes, I don't see why this title is allowed.

> Otherwise please use the original title, unless it is misleading or linkbait; don't editorialize.

> Don't solicit upvotes, comments, or submissions.

Below, I link to the original announcement which was posted to /r/rust. The author talks about the choice of Rust a bunch.

Okay, but my point still stands - that has nothing to do with what was posted to HN. That part was included in the title to get upvotes.

It was included in the title since the author though that the language choice mattered.

It does matter in the context of the author addressing the /r/rust community. If you had posted the reddit link instead this would make sense.

But I don't think the language is relevant for the direct inciteful.xyz site itself. Better to submit both links separately than trying to combine them as they have different audiences.

Posting it twice is a terrible suggestion.

Posting one and linking to the other in the comments, as you did, is the best way.

My first question was how much academic literature written in rust is there.

There may or may not be some interesting reasons for choosing rust, but also the original article does not even include Rust in the title. It seems like Rust is mentioned solely for attention in this case. Would "A better way to search through academic literature" have been popular? Probably. How about "A better way to search through academic literature written in Java"? I assume not.

Regarding moderation, its a thankless task which I don't envy and its hard to draw a line over nitpicky article titles when one has been voted in already.

From an HN point of view Rust is so frequently discussed that it's probably better not to orient a submission like this in that direction—it will probably get sucked in the generic direction, and the diff here (new academic search engine) is more interesting.

https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...

https://hn.algolia.com/?dateRange=all&page=0&prefix=false&so...

Reply on news.ycombinator.com