Frustrated with the limitations of what was available, I did just that.
Turns out they are cheap to build and operate, and and an endless font of interesting engineering challenges.
Turns out they are cheap to build and operate, and and an endless font of interesting engineering challenges.
One side of the problem is to extract keywords from the documents, probably a good idea to start with something along the lines of OpenNLP and use that for TextRank. Well that's roughly what I'm doing now anyway, after having tried a few other approaches. It depends a bit on the nature of the documents I guess. Proper NLP is slow, but dumb TF-IDF goes a long way. (just google these things and you'll figure it out).
If you want full text search, you'll probably want to look into constructing a suffix tree as well. Or just fake it using N-grams, but you absolutely need some rules limiting which N-grams are valid search terms, or their numbers grow astronomically. POS-tagging is useful here if you can afford it.
As for the index: First you need a component that maps keywords to integers. This only needs to be one way, I use a custom memory mapped hashtable, but depending on how much data you've got, there are probably easier alternatives.
Next you need to set up a reverse index that maps words-id to document-id. Again, this is a bunch of memory mapped binary data, I use a mix of hash tables and sorted arrays for binary searches.
That's sort of it.
Then you draw the rest of the owl.
The repo for deploying Solr on Google Cloud is here: https://gitHub.com/kordless/grub-2.0
- HTML parser.
- Crawler.
- Index.
- Constraint solver.
- Query language.
- Ranker.
- Front end.
You can use libraries for as many portions as you like to bootstrap. It's a great personal project.