HNHacker News
TopNewBestAskShowJobs

gioark

7 karma · joined June 6, 2017

software engineer at internet archive

twitter: https://twitter.com/giovannidamiola

submissionscomments
gioark··on Making the Internet Archive’s full text search faster
sure, feel free to contact me by email or DM on twitter. gio archive org
gioark··on Making the Internet Archive’s full text search faster
We don't use stopwords because we want to find all the best and complete matches. We don't want to ignore any of the words part of the search query.
gioark··on Making the Internet Archive’s full text search faster
hi, I am the author of the article.

@devhead: To import the data into ES we used a custom application to extract the text from the OCR'd documents. This is required to support our bookreader software. A complete ingestion takes a few days; we rate-limit indexing in order not to overload the cluster, and maintain reasonable search performance.

gioark··on Making the Internet Archive’s full text search faster
If you are having problems loading the page, try this url: https://medium.com/@giovannidamiola/making-the-internet-arch...