Comment by galkk | Hacker News Reader

Comment by galkk | Hacker News Reader

Parent Full thread

I worked a little bit in the area. (it was 10 years ago in the area of book catalogs matching, per different stores/countries/bestseller lists)

ISBN is a an attribute/key, but not primary key, in database terms :)

ISBNs are messy and in real world you’ll see crazy amount of broken/edge cases that shouldn’t happen by the letter of the standard, but happen all the time in reality.

* For example, isbn can be reused by publisher for completely different book.

* 2nd edition, while very different, may have same isbn.

* Reissue of the same book could have different isbn.

* Textbook of same author for 6th and 7th grade could have same isbn.

* As soon as you’ll get in translations all bets are off.

* I already mentioned textbooks. How anbout about college books where each year there was slightly revised edition of same book.

If you ask yourself - wtf? You’re not alone.

—-

In my youth I heard horror stories about people who suddenly found multiple duplicate guids (uuidv1) in their databases because cheap Chinese knockoff network cards were using same MAC addresses. Think that with isbn that could Happen to you any time.

I did some data collection on my cookbooks. Figured out Lidl had used same ISBN for same book. In entirely different languages.

You feel my pain :)

Honestly, right now I probably wouldn’t even try to code complex algorithm of book matching but fed all of books metadata, including book covers etc to llm and it would do better than what we had.

Our algorithm had tons of special cases coded and in results ui there was a button “needs manual review”, that was launching review workflow (not a joke, business people has special support team in India, because we were matching not only books) for cases when confidence score was low.

But covers also lie. They are mass-uploaded to various services by the similar automated matching systems with cheap human agents used for decision making. Everyone just grabs and re-uploads the existing images from the image search results if they are “similar enough”. Because if it works for instant noodles, it works for books, too. Many publishers don't even bother to provide anything but a single image that is cut from the source file for the printed cover, which might be quite different from the way the book really looks.

For my language, I know I can only rely on just a couple of shops that make a photo of an actual book under regular lights for catalogues and announcements as a matter of principle. Others just add the same image file everyone else used. For the older books that had multiple revisions, it's still strictly manual interaction with second-hand book sites and old listings to figure out the exact version.

On the other hand, most services here do care about providing table of contents, even if its crowd-sourced phone snapshots of the page. International market is insane in that regard. Not only sellers on Amazon and the like ignore them (sure, they were never going to do so much manual work for all the uploaded items anyway), the publishers don't even mention what's inside on their official pages. “Selected stories. New translation by X”. Are those the same stories that were published 5 or 10 years ago under a different name or cover? Has X translated a new set of stories and (partially) refreshed the collection? Both of these things happen. It is ironic that I need to download the full scan from the pirate library to find out what is available in print. Unless, of course, some anonymous worker in some library hasn't diligently typed the whole list of works into the description, hooray for those.

Reply on news.ycombinator.com