OpenAlex: The Promising Alternative to Microsoft Academic Graph | Hacker News Reader

OpenAlex: The Promising Alternative to Microsoft Academic Graph

library.smu.edu.sg

82 pointstokai21 comments

We use this at scite [1] to power some of our features, kudos to Jason, Heather and the rest of the team for their awesome work on maintaining this infrastructure (and the rest of their work on Unpaywall, etc.)

For context, many companies, including us, used Microsoft Academic Graph but it was shuttered last year [2]. So OpenAlex came in to replace, but they also made it backwards compatible with the MAG data, making it super easy for us to migrate.

Overall there are some problems with the data that we've found and reported / fixed (and continue to do so), but it feels well supported to use moving forward.

I believe the OpenAlex team put out a preprint about this recently [3].

1. https://scite.ai

2. https://www.microsoft.com/en-us/research/project/academic/ar...

3. https://arxiv.org/ftp/arxiv/papers/2205/2205.01833.pdf

Thanks for the kudos! I'm very glad we've been able to help out!

Nthing thanks for Unpaywall and any efforts to free people from Google Scholar etc. Arcadia seems to be funding good things.

First I thought having a REST API isn't that "open", but discovered you can mirror the complete dataset (~200G): https://docs.openalex.org/download-snapshot

And the data is licensed under cc0.

We emphasize the API because imho a well-documented, high-throughput API is something really lacking in the ecosystem right now. Dealing with a dataset this size (200M works, 200M authors) is a pain, especially since many end users for this data don't have a lot of technical expertise. Often people have really basic questions (like the ones from the linked post) or they want to build simple applications like monitoring dashboards, recommender systems, and scholarly search engines.

With the API, folks can outsource the heavy data engineering to us for free, and just do the fun parts themselves. We want to make building real-world apps on the global research graph fun and easy, the kind of thing you can do as a hackathon project, instead of with a six-figure grant.

That said, I agree that it's absolutely essential the entire dataset be easy to download and mirror as well. It's called OpenAlex because it's _open_, soup to nuts (the "Alex" part is homage to the ancient Library of Alexandria). All the data is open, the code is open, and our governance is as open as we can make it. [1]

[1] https://ourresearch.org/transparency

Anyone know of any nice front-ends or visualizations for this?

I guess a Python or R Notebook and network analysis libraries would be a good place to start?

I don't know if it's related to this database or does what you're looking for but I've recently learned that https://www.connectedpapers.com and https://www.researchrabbit.ai exist and have started using them (particularly connectedpapers) pretty heavily.

Thanks, both seem like good options!

You can also do visualizations with https://scite.ai! The edges of the graphs show exactly what the papers (nodes) said about each other :)

Here's a video: https://www.youtube.com/watch?v=P5J9EgAaai4

That's pretty cool. It made me suddenly think that it would be great if there was a way to use this safely and without going broke while performing peer-review.

Scholia https://scholia.toolforge.org/ is a frontend to comparable data on Wikidata, a general-purpose knowledge graph. Edit: The OpenAlex website states that a web interface is planned for Summer 2022.

If you go that route, you may also want to check out these community-contributed Python and R libraries: https://docs.openalex.org/api#client-libraries

Jason here, cofounder of OurResearch, the nonprofit behind OpenAlex. I appreciate everyone's feedback! Happy to answer any questions.

OpenAlex seems to be a really useful alternative to Scopus and Web of Science. Although I found that their author-disambiguation is not as good as the one from Scopus.

I would like to see that they make it easier to download database dumps. Their json-based format seems to contain a lot of redundant information. A compressed dump should not be much bigger than 20...30 GB (Theirs is 200GB, according to https://docs.openalex.org/download-snapshot/download-to-your... )

Agreed, the author disambiguation isn't quiet as good as Scopus'...they have a bit of a head start on us. But we're improving it quickly.

Thanks for the suggestion about the data dump. A lot of that weight is abstracts, which come in at over 30GB just by themselves. But it's true that the JSON format has some redundancies. For now we think those are worth it, because the denormalized schema is very compatible with the API and easy for beginners to get started with. Plus you only have to download it once (for free! HT to AWS Open Data sponsorship), and after that the updates are very light.

We'll certainly consider offering a smaller, normalized format in the future though, if we get more requests for it.

Author disambiguation can be a very hard problem in practice, especially with authors who share the same names (a relatively common occurrence in some non-Western countries) and work on similar things. It's even somewhat common to see the exact same ORCid (supposedly a unique identifier) attached to what's clearly papers published by distinct individuals. At some point, one pretty much has to guess.

Yes, I think some services attempt to retro-assign ORCIDs. I usually have to try to grab the affiliation and date to make sure I have the right person. Someday for the authors I follow I want to train a model to give me a match score with my Smiths and Lees etc.

This looks very similar to what meta.org developed before it got shut down to pave the way for the Facebook renaming. I wonder if SMU inherited meta.org's archives and data pipelines.

Meta only focused on biomedical information. This is explicitly based on Microsoft Academic Graph which was all encompassing. They did remove patents, tag papers with fewer topics/subtopics, and don't index as many conferences, but they did add a couple of nice things like ORCID. [0]

[0] https://docs.openalex.org/download-snapshot/mag-format/mag-m...

To be clear, this is a project isn't affiliated with SMU, they just did the blog post...it's from a nonprofit called OurResearch (source: I'm a cofounder).

We did have some good conversations with folks at Meta before they closed up shop, but didn't end up using any of their data.

Reply on news.ycombinator.com