Hacker News Reader

Comment by skizm | Hacker News Reader

I always wonder, how do sites like this get their list of books and book metadata? Do publishers have an API? What about Amazon?

View on HN

It is so bad!!!

I use Nielsen's API, but the data is pretty rough, and you have to spend a lot of time cleaning it. Plus, the archaic industry standards around genre are hard to translate to what readers use - https://www.bisg.org/complete-bisac-subject-headings-list.

Ingrams and Bowker are the other big metadata providers. Ingram's is good but expensive, but the data faces the same issues.

If you have an ISBN you can also check Worldcat, but at scale it's also probably not free. And if you're working with anything that might have an academic slant, Crossref can be useful.

Book metadata is very challenging. Even the publishers of said books are pretty bad at delivering good metadata.

WorldCat doesn't have an official API for anyone but member libraries, scraping the site is not fun, and the quality of the data isn't great (it's essentially bulk-imported from member libraries' catalogs).

National databases like the Library of Congress are significantly better - WorldCat is best used as a fallback for books that aren't included in the high-quality databases.

Ya I do pull in LOC data, but its limited to a very small number of books. Good quality though.

I forgot if Anna's archive contains metadata but that'd be my free-ish solution

I thought Anna’s was dead?

The last blog post is from just a couple days ago. They're apparently setting their sights on a map of all known ISBNs, and hoping to turn the map from red to green

Do any of the sites use collaborative filtering for “uses who like x also like y” for a nice filter bubble?

I imagine it complements or my even supersede tags.

(Admit I haven’t looked at all the sites people are mentioning in the comments yet- lots of good leads!)

ya, I do this; I am not sure what the other websites are doing, though. If you email them I bet they will tell you.

Interesting that Wikipedia considers "campus" and "varsity" to be novel genres, but neither of those words appears here on BISG:

https://www.bisg.org/fiction

https://openlibrary.org/ has a pretty good set of data and a decent API. You can mix and match too, since openlibraries covers kinda suck half the time.

It's alright. I would use it for a tool where the most important part is having a more-or-less accurate author <-> title <-> ISBN mapping, but not for anything where I need precise bibliographic metadata.

You want to read a screed about this? I just published bc I couldn't figure out how to share unpublished substack post, but it needs to be edited: https://bookheadlife.substack.com/p/the-mess-of-bibliographi...

Freebase got about 50% of the way to a good book database but Google killed it.

Reply on news.ycombinator.com