The DOI System | Hacker News Reader

The DOI System

doi.org

64 pointsgone3565 comments

DOI is one implementation of handle system defined in RFCs 3650, 3651 and 3652.

"RFC 3650: Handle System Overview" https://www.rfc-editor.org/rfc/rfc3650.txt

"RFC 3651: Handle System Namespace and Service Definition" https://www.rfc-editor.org/rfc/rfc3651.txt

"RFC 3652: Handle System Protocol (ver 2.1) Specification" https://www.rfc-editor.org/rfc/rfc3652.txt

Digital Object Architecture and the Handle System - icann https://www.icann.org/en/system/files/files/octo-002-14oct19...

>The Digital Object Architecture (DOA) is an overall architecture for managing digital objects with an associated unique persistent identifier. In this context, digital objects are defined as a sequence or set of sequences of bits. The DOA resulted from the work by Dr. Robert Kahn and Dr. Vinton Cerf at the Corporation for National Research Initiatives (CNRI) in the late 1980s.

>The DOA has three core components: the identifier/resolution system, the Digital Object Repository system, and the Digital Object Registry system containing metadata about the repository objects. The Handle System is the original name of the identifier/resolution system of the DOA. Its governance is coordinated by the DONA Foundation, a Geneva-incorporated nonprofit organization founded by CNRI.

>The Handle System and the DONA Foundation are the elements of the DOA that are closest to ICANN organization’s role helping coordinate the Internet's system of unique identifiers. This report will focus on those two elements to better understand the technology and its usage, innovation, and limitations. This report is not intended to endorse the technology nor offer any recommendations regarding its operation. It is based on the set of publicly available technical documents, an analysis of the code published on CNRI website, and a number of interviews with Dr. Robert Kahn and his team at CNRI.

https://www.handle.net/

Visit their old demo pages from early on (5 years into their existence though, after they had time to work out the kinks) where they had examples of DOI links that supposedly will never break:

https://web.archive.org/web/20050305141724/http://www.doi.or...

Don’t just follow the links as they are on archive.org, because that wraps the links and you get an almost comical series of ultimately unsuccessful redirects (unlike many redirects on archive.org which often ARE successful when not using DOI links).

No, try the DOI links themselves by copy pasting them without the archive.org wrapper. I think you’ll find many of them (most of them?) are not working. This version of the page is from after “the handle system” had been up and rolling for a few years.

To be fair, maybe some browser plug-in is needed?

Notice also how thickly the web site is festooned with copyright, trademark, and registered trademark notices and the like. Feels off to me.

> examples of DOI links that supposedly will never break

I looked in the FAQ and related materials from that 2005 mirror, and didn't see where they said the links would never break. Did they promise to archive every resolvable DOI in case a registration agency should go out of business or stop being a registration agency?

FWIW, here's what I found:

doi:10.1038/35057062 - works

doi:10.1002/0470841559.ch1 - works

doi:10.1228/0103000001002 - does not work. Neither DDG nor Google Scholar know of the whitepaper "Secure Internet Delivery of High Value Content". The source organization, "Lon Inc." does not seem to exist, or at least, http://loninc.com/ does not resolve.

doi:10.1228/0103000001009 - does not resolve. Also from "Lon Inc."

doi:10.1430/8105 - resolves to https://www.medra.org/servlet/MREngine?hdl=10.1430/8105 resolving in turn to https://www.rivisteweb.it/doi/10.1430/8105 .

doi:10.1392/BC1.0 - works

doi:10.1786/SIBQK6AD8RNA - does not work. The corresponding document appears to be https://www.legislation.gov.uk/ukpga/2003/4/section/4 , with no associated doi.

The other four DOIs from The Stationery Office UK Official Publications Catalog also do not resolve.

While I can find the press release notice when TSO applied DOIs to UKOP (at http://www.doi.org/news/TSO-UKOPPressRelease.pdf ), I cannot find why they decided to no longer use or support DOIs.

It looks like one registration agency went out of business and another stopped supporting DOIs.

What should DOI have done to improve things to your satisfaction?

Sci-Hub works well with DOIs, and without festooned notices and the like. ;)

Discussions about problems with the DOI system and suggestions for potential replacements for it come up every few months. So a few years ago, I wrote something that helps outline some of the issues one would need to address:

https://www.crossref.org/blog/rehashing-pids-without-stabbin...

On the odd chance that someone needs a JS library to detect whether a webpage exposes a DOI that identifies it (eg for a browser extension for academics), I wrote one that might be useful: https://www.npmjs.com/package/get-dois

That would be super helpful when I find a journal article that's on Pubmed, but doesn't have a DOI attached to it anywhere that I can find.

Anyone wanna take on this project? I already use:

Open Access Button: https://openaccessbutton.org/ UnPaywall: https://unpaywall.org/

And Sci-Hub when I have too, but sometimes I just can't find the DOI on some articles.

I believe the one big issue with DOIs is they rely on the resource owners to update the centralized database if the resource location changes. So links can still break/rot despite knowing the URN.

Another issue is that it doesn't guarantee the URN (i.e. DOI number) uniquely identifies a resource. So a single resource can have several aliases, potentially confusing discussions or bypassing blacklists, etc.

Can someone tell me where to get a definitive list of DOI prefixes? I tried to find this some time ago and came away very frustrated. The system seems in reality very opaque.

DOI prefixes are designed to be opaque. All they do is serve as a namespace that helps guarantee that somebody can generate a unique DOI (yes, I know, other approaches to this would arguably be better). In short- the prefix doesn't really tell you anything useful about the DOI. So what are you trying to do?

And have you looked at the Crossref and/or DataCite APIs?

https://api.crossref.org https://api.datacite.org

Shout out to all the Crossref/Datacite people in this discussion. I use your services in polite mode for discovery and don’t know what I’d do without them. If any ORCID people are in the house you get some appreciation too.

I struggle to link articles to datasets but if the solution is out there it is probably based on your work.

What are these for exactly? I'm looking for a paper that doesn't have a DOI attached. Would there anyway to use these tools in a Windows environment to search for the DOI?

Worth reading the Wikipedia article on the Handle System, of which the DOI system is a subset:

https://en.wikipedia.org/wiki/Handle_System

All handles that start with 10 are DOIs. Handles that start with 20 are issued by CNRI.

I don't know of a published list of prefixes, but it would be easy to investigate since entering the prefix into the appropriate resolver will bring up information on the issuing authority:

- https://hdl.handle.net/20.1000 : "Prefix for CNRI licenses."

- https://doi.org/10.14288 : "Datacite prefix 8/22/13"

Not all have the DESC field, but you can figure out who these are from other fields:

- https://doi.org/10.14288 : CrossRef & American Chem. Soc. publications

- https://doi.org/10.1038 : CrossRef & Nature publications

etc...

> All handles that start with 10 are DOIs. Handles that start with 20 are issued by CNRI.

Interesting! I manage a system that is responsible for two Handle prefixes: 10568 and 10947. Guess we got those from CNRI around 2009 and it was different then.

haha, yeah, keep peeling the onion https://handle.net/

Don’t know why you are downvoted, this site has always popped up as one of the big stakeholders in the weird DOI schemes.

This is the equivalent to tracing a domain registrar up to ICANN.

Yes, which is a significant connection to make when understanding the big picture here.

I’ve never understood this as anything other than an oddly secretive power and money grab by some entrenched stakeholders in publishing and document delivery.

They seem to have non-guessable, non-hierarchy-revealing URLs that must be resolved using a database they control.

It strikes me as a showcase of ways to try to get around web standards to make things locked down.

Last time I looked at this (with horror, not with interest), the fees to become a member or to register links were exorbitant and the system itself seemed to be geared around supporting payment and DRM scenarios while making promises to fix non-existing problems (or ones that could be fixed for free using well known standard web practices).

> non-hierarchy-revealing URLs that must be resolved using a database they control

What do you mean? There is a journal prefix (Nature has 10.1038) and the link usually translates fairly directly to a URL for new articles.

EDIT: Just looked it up in the DOI rules. The registrant chooses the suffix. So the journals are free to maintain 1:1 mapping to URLs if they wish. I assume DOI is popular because people trust that the central entity might do a bit better job with regards to link rot than the journals themselves.

To take your point to the next step, how do two otherwise unrelated journals (of which there are tens of thousands) link between each other and solve link rot? Answer: shared open non profit metadata infrastructure.

Disclaimer - I’m at Crossref AMA.

Theoretically, the journals could publish a canonical URL for each article. But apparently organizations are not good at publishing and maintaining such URLs. Especially in case of mergers and rebrandings.

That canonical URL is literally what the DOI system is. Publishers getting together to agree on a shared identifier system, regardless of each publisher's business model.

Compare the readability and editability of:

nature.com/2021/11/17/news/

versus:

10.1038.123.366.345643

And you ask what I mean. What I mean is the first one is transparently better in several ways.

If I got the syntax wrong, just focus on the first part then, nature.com versus 10.1083. One is more readable than the other.

Neither your URL nor your DOI resolves. I meant rather the correspondence of

    https://www.nature.com/articles/1771046b0

    to

    10.1038/1771046b0

    https://www.nature.com/articles/s41586-021-04114-w
    
    to
    
    10.1038/s41586-021-04114-w

Look at the metadata of those two articles.

    https://www.nature.com/articles/1771046b0
    has
    <meta name="dc.identifier" content="doi:10.1038/1771046b0"/>

and

    https://www.nature.com/articles/s41586-021-04114-w
    has
    <meta name="dc.identifier" content="doi:10.1038/s41586-021-04114-w"/>

They aren’t meant to resolve, but to show the difference between friendly and unfriendly ways to construct a URL (or URI if you prefer).

But yes that’s interesting and good to know that sometimes they can choose to use corresponding components in the URL versus DOI.

>... Compare the readability and editability

Indeed, I was wondering why not to adopt an approach similar to a Git repo where each leaf (document) is identified with a ref hash.

It could be used with a publisher prefix to segment the hash-space.

Basically, it's a repository of metadata. Each publisher could equally host the whole repo, but extend only their corresponding sub-repo.

The central entity could then maintain the publisher's name-to-id convenience xref and some search facility.

> 10.1038/nature.com/2021/11/17/news/xxxxxxx

can be a valid DOI, 10.1038 has the same role as the nature.com domain but with a different governance

That’s why the display guidelines stipulate that DOIs are always expressed as URLs.

Sure, but 10.1083 is still a shitty link compared to nature.com, independent of which url prefix is used.

There are different timescales involved.

The resolvability of a DOI is obviously important, and you have found a few examples where the steward of the content (ie publisher) hasn’t updated the links. But in that case DOIs can in theory be redirected to archived versions. That's the price you pay for a large, diverse community of publishers. Don't imagine that publishers would magically be any better at fixing their own websites.

But more broadly, as identifiers, having an agreed scheme helps link publications, datasets, other entities such as funders and institutions. It allows discovery of metadata in a way that’s discoverable by all.

Someone will want to follow those links in 100 years and whatever state DNS or WWW is in, the metadata will be in an open archive somewhere.

(I'm at Crossref, AMA)

Taking you up on your AMA, is there a best practice for discovering datasets belonging to articles and vice-versa? I see that Web of Science has a lot of this but they don't make that linkage explicitly available as an export. I have tried querying isSupplementTo and chasing cited works, and I don't expect a smoking gun.

Just now I tried searching Scholix via the web interface for IDs that are linked in Dryad, which is supposed to use the same linking system, and in either direction came up empty.

TIA for any clues!

Scholix[0] is a collaboration between a few orgs, including Crossref (DOI registration agency for scholarly content, e.g. articles) and DataCite (DOI registration agency for data sets).

If a DataCite member registers a data set, and mentions a link to a Crossref DOI, that's almost certainly a dataset-article link. Vice versa.

So the data in Scholix is the union of "what citations do article publishers and dataset publishers think exist".

We're working on improving how we process the data citation data, but ultimatley can't improve on what Publishers provide.

Sorry, a bit of a non-answer.

[0] https://www.scholix.org/

The https link doesn't work for me, working link: http://www.scholix.org/

Journals, even prominent ones, change names for all sorts of reasons. It doesn't happen often but it does happen. The meaningfulness issue can cut both ways.

I think the real problem is the centralized vs federated vs distributed nature of it. IPFS is a good example of how that could have looked; not sure if it could be moved into that space somehow (I'm sure it could in theory, but in practice?)

Dois are not supposed to replace urls. They serve very different purposes. A doi is a unique identifier for a scientific article, that’s all it is.

The sole purpose of DOIs is that they never change. The moment Nature decides to update their CMS, the first link is going to 404.

As is the second link, as many DOIs have.

> I assume DOI is popular because people trust that the central entity might do a bit better job

First, I don’t know how popular it is.

Second, look at what an absolutely poor job the doi.org site themselves have done keeping their showcased example links from their demonstration document from breaking over the years. See my comment elsewhere here about what I found on archive.org. What am I missing here? It looks like their link rot is close to 100%.

> First, I don’t know how popular it is.

Very. The doi is now the unique key to identify a scientific article.

Researchers like it because it is an easy way to get canonical metadata without having to go through Google Scholar or (dog forbid) something clunky like Science Direct. You just put the doi and bang, you’ve got the article. No issue with a typo in the volume number, or authors who can go by different names (or the same name written differently), or anything like that. It has make managing bibliography databases massively easier.

Editors like it because it’s a very efficient way of checking the information in the bibliographies of submitted manuscripts.

I have no clue about their examples, but in the real life I have never had a link rot issue in ~10 years handling thousands of references.

Easily fetching the canonical metadata is so useful when doing a literature search and collecting sources, or trying to put together a bibliography. In Zotero (and presumably other bibliography managers), you can just plug the DOI in and you instantly have the title, authors, abstract, etc.

DOIs are not URLs though. They are identifiers, as the name suggests. I don't think URLs were ever intended to be identifiers. At least that's not how they are used, as can be seen from the fact that there is a many-to-many relationship between URLs and pieces of content. Identifiers for published pieces that are independent from where that piece is currently published are useful. Just one example: An article can be published in many places and in many different formats (HTML, print, PDF, ...). The DOI allows us to establish that all these instantiations represent the same publication. You might say that we could use titles instead but these are not unique. We also can't use title + authors because authors change their names. And so on. DOIs therefore address a real need. That's why they've become so successful and omnipresent in academic publishing.

URLs evolved into URNs and URIs

I dunno, looks similar to the ISBN system for books.

ISBN = International Standard Book Number

DOI does resemble ISBN, but I can find book info via ISBN without relying on the publisher to maintain a database.

(via Google or ISBN.nu or maybe even the Library of Congress)

ISBN is indeed assigned by the publisher, who has been granted an ISBN prefix by the ISBN Fairy. That's same as DOI.

But DOI feels more like Amazon's book numbering system, their ASIN. That relies on an opaque entity which may well lose track of any particular item, for their own reasons... though ASIN has not been promoted as a standard index.

Here's a really old ISBN, which I picked at random because I wrote the book (often also a random process at times) -- almost 25 years ago:

978-0782120813

I had no trouble finding the book via this magic number. I don't know how that compares to a 25 year old DOI for an item that was in limited publication.

(shameless plug, fair, but it's an example for which I happen to know the provenance of the isbn, even before publication, and I also have an idea at how obscure this item turned out to be after public distribution.)

Open metadata records dump: https://www.crossref.org/blog/new-public-data-file-120-milli...

DOI Content Negotiation to retrieve metadata in a way that's interoperable between DOI registration agencies: https://citation.crosscite.org/docs.html

Crossref is expressly not opaque. Although we have a legacy codebase, we've committed to fully open source and open data.

https://www.crossref.org/blog/crossrefs-board-votes-to-adopt...

(Disclaimer - I'm at Crossref)

It would be awesome if there actually was a reliable, API-accessible (and reasonably cheap/free for small numbers of automated lookups) database of ISBNs. I want to build my own book inventory system and be able to look up ISBNs in my own tool.

Dare I say it… this domain seems like a good use of blockchain. Content won’t go away, references can be baked into the graph, makes preregistration explicit, edits would be transparent.

IPFS (and other technologies like it) are essentially just distributed DOIs, which are extremely functional and have been used for years, yet get very little appreciation in tech field.

The "just distributed" is the interesting bit though. DOIs are really the same as URLs, except the lookup doesn't use DNS.

I think DOI is essentially a URI schema. I don't think there is a technical reason preventing them from being distributed like IPFS, in fact the underlying resources are distributed it's just the URL lookup that is centralized.

Yes, it is a URI schema. FAQ #11, at https://www.doi.org/faq.html

> DOI & URI: how does the DOI system work with web URI technologies?

> DOI names may be expressed as URLs (URIs) through a HTTP proxy server. In addition, DOI is a registered URI within the info-URI namespace (IETF RFC 4452, the "info" URI Scheme for Information Assets with Identifiers in Public Namespaces). See the DOI Handbook, 2 Numbering and 3 Resolution, for more information.

I know little about IPFS. I know a bit more about DOIs.

A DOI maps more to an abstract, mutable resource than a specific document or file. That doesn't seem so easy to express in IPFS.

Take doi:10.1093/nar/gkz173 which I picked arbitrarily. This resolves to https://academic.oup.com/nar/article/47/11/e63/5377471 .

That shows the content of the paper in HTML, with an option to download in PDF.

It also has references to the journal issue, methods to get the citation in various format, a navigation bar for the site, and branding for the journal.

Many of these may change over time. Indeed, the paper itself may change if there are any updates or corrections, without changing the DOI.

Or the paper might be retracted, like doi:10.1126/scisignal.abn0168 which resolves to https://www.science.org/doi/10.1126/scisignal.abn0168 . The original paper is no longer accessible. (An alternative is to add the equivalent of a big red stamp on it saying "RETRACTED" while leaving the content accessible.)

Finally, I suspect many journals would not accept a scheme which prevents them from charging for access.

Could you explain how IPFS might handle these issues of multiple formats, mutability, and payment?

> The original paper is no longer accessible. (An alternative is to add the equivalent of a big red stamp on it saying "RETRACTED" while leaving the content accessible.)

I think you are slightly confused - the "alternative" that you describe is actually what happened in this case (and in all the cases of retracted papers that I've encountered).

A Retraction Notice has been published at a separate publication [1] with its own DOI, and it cites the original paper [2].

The original paper [2] has been updated to indicate that it has been retracted. In the HTML version, there is an orange banner saying "This article has been retracted". In the PDF version [3] there is a red header, and the retraction notice has been appended as an extra page.

But you are correct that it is important to be able to update PDFs if a significant mistake (or deliberate fraud) is identified.

[1]: https://www.science.org/doi/10.1126/scisignal.abn0168

[2]: https://www.science.org/doi/10.1126/scisignal.aag1064

[3]: https://www.science.org/doi/epdf/10.1126/scisignal.aag1064

Indeed. I misread the retraction notice quite severely.

Thank you for the correction.

As an alternative example, I searched Retraction Watch.

https://retractionwatch.com/2021/09/23/alzheimers-diagnosis-... describes the paper:

> “Intracranial pressure waveform changes in Alzheimer’s disease and mild cognitive impairment” — which now seems to have disappeared entirely from the journal’s site — appeared in Surgical Neurology International in July.

It points out the paper is still in PubMed Central, at https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8168790/ . However, its DOI, 10.25259/SNI_48_2021 , redirects to a 404 page from the journal.

Well, you could still use the multi-hash thing for the identifiers without distributing them on IPFS,

though it would make it much more convenient for people to pirate them, because then people wouldn’t have to look up “what is the multihash for the thing with this doi?” before requesting it on ipfs.

re: mutability you just have the standard be that the document being linked to is a sort of wrapper to the actual one. My guess is this is really what's going on with doi anyway.

But you're correct that the mutability issue is sort of a tricky one.

There's this from IPFS about mutability:

https://docs.ipfs.io/concepts/file-systems/#mutable-file-sys...

Since we're on this topic, anybody knows a way to get (very) cheap DOIs? I know people who try to maintain independent repositories of academic papers, and are successful at it, but don't have the budget to get standardly-priced DOIs for all those articles - one of the few complaints their users have.

It depends what you're publishing, but if it's fully open access then you can post it at https://osf.io and get a free DOI almost immediately. Obviously it's "their" DOI. If you want your own prefix, then you need to register as a DOI issuer (or whatever the correct term is) which I presume is the cost you mention.

To me it seems similar to certificate authorities. There are several DOI issuers who act a bit like letsencrypt, doing their best to make an opaque process easier to navigate. But it simply takes a fair bit of cash to become an issuer.

Maybe there's space here for a different approach, but DOI and Handle are the incumbents and the "market" is resistant to change.

Have a read through the posts at the Crossref website, including the minutes from their meetings. There's plenty written on HN too about the likes of Elsevier and their monopolization of academic publishing.

Thanks! Wrt "There are several DOI issuers who act a bit like letsencrypt" -- I haven't seen any that are free like letsencrypt, did you have in mind any concrete issuer?

Huh I forgot that DOIs are meant for more than just academic journals.

Do DOIs still make sense if we have web3? Would the future of DOIs be similar to a filecoin kind of system, instead of using resolvers and organizations like CrossRef?

Still not sure if we'll have web3. It kind of seems like a great way to suddenly make 50% of web infrastructure carry nothing but transaction data from node to node, and when you think about it with private data storage it makes laws like the GDPR unenforceable so there might be considerable state backlash. Well, we'll see how it develops.

A more important concern is whether we need DOIs with the current W3C standards for Semantic Web. We already have a multitude of absolutely giant databases such as Wikidata, but also many others linking together semantic information from all over the web into a giant machine and human readable network: https://www.wikidata.org/wiki/Q686797. Do you really need an incomprehensible centralized system such as DOIs when you have a multitude of separate incomprehensible IDs linked together by human readable semantics?

Better known as “those things you paste into Sci-Hub”.

unbelievable that this has not been superceded after so many years

DOI's allow Sci-Hub to function pretty well. One good example of how they have found a purpose to serve no one else was doing.

DOI's also apply to datasets ie. - https://www.ands.org.au/guides/doi

DOI's can theoretically apply to paragraphs, sentences, words. Each having a different DOI. There's a lot to them.

The in practice case for DOI's seems to be like a approximate hash function for things, with the location included. From memory retracted articles will have the same DOI, even though it will be different since it'll say it's retracted.

Reply on news.ycombinator.com