Hacker News Reader

Comment by perihelions | Hacker News Reader

- "To complete the download of the full dataset, please fill out this form, which helps us report usage back to our funders, and we'll immediately provide you with a download link."

https://unpaywall.org/products/snapshot

Is that dataset different from this un-gated one? They're both indexes of Crossref DOI's, and they're both 120 million records long.

https://www.crossref.org/blog/new-public-data-file-120-milli...

View on HN

Unpaywall is based on Crossref DOIs (one-to-one records), and adds information about publicly accessible versions of each work. In theory publishers can register metadata about whether articles are OA or not with Crossref, but the quality and coverage of this metadata is poor in general.

Unpaywall will both check if articles are actually available from the publisher (by following the DOI and parsing the landing page), and by looking for other versions elsewhere on the web (eg, a pre-print). It is simple in theory, but doing this reliably for millions of DOIs from thousands of publishers is a lot of work!

Unpaywall is different to crossref. The size will be similar as a lot of scientific DOIs are from crossref, and they're a common dataset to base initial data on.

https://unpaywall.org/data-format is the format for unpaywall

https://api.crossref.org/swagger-ui/index.html for crossref info

Would not be surprised if the Crossref DB was their starting point.

It is indeed :) (COI: Unpaywall cofounder and dev)

Reply on news.ycombinator.com