Comment by gary_0 | Hacker News Reader

Comment by gary_0 | Hacker News Reader

Parent Full thread

> To make the most of phrase matching, stop words need to go.

Perhaps I am misunderstanding; does this mean occurrences of stop words like "the" are stored now instead of ignored? That seems like it would add a lot of bloat. Are there any optimizations in place?

Just a shot-in-the-dark suggestion, but if you are storing some bits with each keyword occurrence, can you add a few more bits to store whether the term is adjacent to a common stop word? So maybe if you have to=0 or=1, "to be or not to be" would be able to match the data `0be 1not 0be`, where only "be" and "not" are actual keywords. But the extra metadata bits can be ignored, so pages containing "The Clash" will match both the literal query (via the "the" bit), and just "clash" (without the "the" bit).

One of the problems with stop words is that they vary between languages. "The" is a good candidate in English, but in Danish it just means "tea", which should be a valid search term. And even in English, what looks like a serious stop word, can be an integral part of the phrase. "How to use The in English".

It's not as bad as you might think, we're speaking dozens of GB across the entire index.

I don't think stopwords as an optimization makes sense when you go beyond BM25. The search engine behaves worse and adding a bunch of optimizations makes an already incrediby complex piece of software more so.

So overall I don't think the juice is worth the squeeze.

Removing stop words is usually a bad advice which is beneficial only in a limited set of circumstances. Google keeps all the “the”: https://www.google.com/search?q=the

I don't think it's as simple for them as just keeping the thes. It's probably very context dependent

in reality, terms are stored with the position, so if you use stopwords

to be or not to be should be indexed as 2-be 6-be and a phrase match should match exactly. the problem would be that it would match also "be not to or be", as the distance between the 2 "be" is also 3.

Long time ago it was necessary, but nowadays you loose more that what you gain using stopwords.

Reply on news.ycombinator.com