Isn't this a concern as the main objective of search is to provide accurate results?
Isn't this a concern as the main objective of search is to provide accurate results?
This should be a concern to the author, but there's no reason to think the search ranking is not working as documented.
[0] http://www.postgresql.org/docs/9.4/static/textsearch-control...
If it provides 99% of the results and misses some documents because of some weird bug or encoding issue, then it could very well be good enough for their purposes. Heck, even 90% could be good depending on what they do (e.g. serve articles in an online site).
For other uses, like police or medical records obviously they'd need 100% results.
Take your favorite language detector, like cld2. Apply it to some real-world language, like random posts on Twitter. Did it detect the languages correctly? Welp, there goes that idea.
(Tweets are too short, you say? Tough. Search queries are shorter. You probably aren't lucky enough for your domain's text to be complete articles from the Wall Street Journal, which is what the typical NLP algorithm was trained on.)
Stemming will always be difficult and subtle. It's useful but it isn't even linguistically well-defined, so you'll have to tweak it a lot. If stemming seems easy, you haven't looked at where it goes wrong for your use case.