Hacker News Reader

Comment by softwaredoug | Hacker News Reader

This is really cool. I have a pretty fast BM25 search engine in Pandas I've been working on for local testing.

https://github.com/softwaredoug/searcharray

Why Pandas? Because BM25 is one thing, but you also want to combine with other factors (recency, popularity, etc) easily computed in pandas / numpy...

BTW phrases are the hard thing. There are a lot of edge case in phrase matching. Not to mention slop, etc. And you want to compact positions into as little memory as possible

https://github.com/softwaredoug/searcharray/blob/main/search...

View on HN

Thanks for your comment Doug! I just bought your book (Relevance Search) and I'm planning to read it asap.

I'll give a look to your project and try it for experimenting, thanks for sharing.

I also bought the book (it’s Relevant not Relevance BTW), that one and Deep Learning for Search were invaluable when building a news search engine a few years ago (right before and in the middle of the pandemic).

Can confirm this book is excellent.

It's one I point people at all the time when they ask me why something isn't working as expected in any standard search tool, and something I reference from time to time to refresh my own knowledge.

Well worth the money.

Thank you! It's a bit old now, but I think the principles in it are still valid :)

Hey, I tackled phrase matching in my toy project here: https://github.com/vasilionjea/lofi-dx/blob/main/test/search...

I think I tested it thoroughly but any feedback would be appreciated!

Edit: I delta-encoded and base36-encoded the positions

Oh nice I’ll take a look!

Have you found that including sentiment analysis has helped(or hurt) with phrases? I also have found that phrases are difficult to work with and I’m wondering what I can do to improve performance.

How come you found this post and commented on it so quickly, do you have some sort of search scanning frontpage for terms of interest or was it by chance? Curious

Huh? I check Hacker News multiple times a day - it's not odd to click on an article within an hour of it being posted.

It's not the first time I saw an article posted and then an expert in the field comment on it rather quickly, I thought I may be missing something how other people use this site, had no negative intentions asking this and thanks for the answer ;)

HN has an RSS feed [0] so there's no need to keep refreshing or make the rounds of this and other sites that have interesting information.

I have my own feed setup with sites I like to frequent [1].

[0] https://hackaday.com/blog/feed/

[1] https://mechaelephant.com/feed

what software did you use for your own feed?

Various scripts in python, shell and javascript [0].

[0] https://github.com/abetusk/www.mechaelephant.com/tree/releas...

thanks! I'm currently looking at the files

how did you filter which posts appear in the feed?

EDIT: I was able to make it work, I replaced the node module xmltojson with xq (python lib available from pip)

https://static.nani-so.re/feed/ many thanks!

Awesome!

There's an RSS feed https://news.ycombinator.com/rss

Some people set up something like a Google Search Alert for terms about them or their writings, etc.

Also, a lot of subject matter experts hang out here so it just happens a lot :)

https://f5bot.com is a way to get an email notification when a keyword is mentioned

I got that HN gaaaame

Reply on news.ycombinator.com