Hacker News Reader

It seems like it uses postgresql's FTS, which will generally drop stop-words so "the", "a", "and" and similar words are dropped. I've been meaning to figure out the best way to deal with this myself, and I'm guessing looking for exact matches first and then running a FTS query could work.

View on HN

You can write a custom stemming algorithm and load it as an extension library into Postgres, then use that with `CREATE TEXT SEARCH DICTIONARY` to create a custom dictionary. It's not as difficult as it sounds - you can use the default Snowball stemmer as a sample, and tweak it.

It's not just a custom dictionary. Stop words are usually excluded for a reason, you need to understand the nature of the query, and when to exclude what looks like stop words from being pruned. It's not really a job for a snowball stemmer, as you do need to operate over multiple tokens to gather context.

Most of the time, keywords like this come from external anchors as well, which is something that you're gonna be able to leverage with this design (as I understand it).

Reply on news.ycombinator.com