Hacker News Reader

Comment by lettergram | Hacker News Reader

I actually built a search engine back in 2018 using postgresql

https://austingwalters.com/fast-full-text-search-in-postgres...

Worked quite well and still use it daily. Basically doing weighted searches on vectors is slower than my approach, but definitely good enough.

Currently, I can search around 50m HN & Reddit comments in 200ms on the postgresql running on my machine.

View on HN

Nice – looks like the ~same approach recommended here of adding a generated `tsvector` column with a GIN index and querying it with `col @@ @@ to_tsquery('english', query)`.

Yeah my internal approach was creating custom vectors which are quicker to search.

Offtopic, but currious what are your use cases when searching all HN and reddit comments? Im at the beggining of this path, just crawled HN, but what to do with this, still a bit cloudy.

I built a search engine that quantified the expertise of authors of comments. Then I created what I called “expert rank” that allowed me to build a really good search engine.

Super good if you’re at a company or something

https://twitter.com/austingwalters/status/104189476543920128...

Wow, that lead me down quite a rabbit hole, impressive work.

If you are just interested in searching HN and don't need full data dumps, the Algolia search for HN is quite good and fast:

https://hn.algolia.com

They also have a free API.

If you need data dumps, maybe look into Google BigQuery.

Reply on news.ycombinator.com