Show HN: I put PubMed in a vector DB | Hacker News Reader

Show HN: I put PubMed in a vector DB

pubmedisearch.com

97 pointsmpmisko27 comments

Hi HN,

As a researcher, I often found myself struggling with the limitations of keyword-based search when exploring PubMed papers. To address this, I created PubMed Search (https://www.pubmedisearch.com/), a tool that leverages a vector database to enable semantic search across medical research literature.

Some key features:

* Daily updates to ensure access to the latest articles

* Semantic search using latest & greatest embedding models

* Some additional useful info about the papers (tldr, journal, publication date, etc.)

Hope you find it useful!

Open articleView on HN

Hey mate, should search by PMID work? Like 35982160 is PMID for "Rare coding variation provides insight into the genetic architecture and phenotypic context of autism" - not seeing this publication at all in search results...

Hi, it currently does not support search by PMID. But you can find the paper included in the results here (5th place):

https://pubmedisearch.com/share/Do%20some%20individuals%20wi...

Very cool!

Related: the NIST TREC (Text REtrieval Conference) has had several competitions over the years related to improving the searchability of medical data: https://www.trec-cds.org/

If you have novel ideas in this area, you should consider participating. https://trec.nist.gov/

Thanks! Looks quite relevant

This is very cool! 2 questions spring to mind:

1. How much did it cost to embed all those vectors and how many articles did you process? PMC is quite large.

2. Could elaborate a little more on your approach to ranking articles? Because I'm familiar with semantic search via embeddings put did you weight those with impact factors/citations? Like how does one even calculate that?

Anyhow, love the idea.

1. We cover all the articles on PMC. The exact cost is hard to estimate because we did a lot of iterations.

2. We do weight those ... it is a lot of trial and error and you have to have good & exhaustive benchmarks.

Congrats on shipping!

I'm curious how the search results rankings work, doesn't look like it's based on date or number of citations, but seems to be deterministic (persists over multiple searches). I did a keyword search using one word.

Thanks!

It uses a vector search approach. Your query is embedded in a vector space using a language model and we find the closest vector to the query from the PubMed papers. This is a good summary of the techniques: https://learn.microsoft.com/en-us/azure/search/vector-search.... There are a couple more tricks but this is the gist.

The nice part is that this approach allows you to find relevant papers to your question. E.g, you can ask "Can secondhand smoke cause AMD?" and the very first few papers are answering your question (https://pubmedisearch.com/share/Can%20secondhand%20smoke%20c...). The more specific question, the better. :)

Cool idea!

What are some papers labeled "High Quality Article"? How do you determine that?

Just looking at stuff like citations and impact factors of journals.

Nice!

Out of curiosity what model(s) are you using to generate the embeddings?

Glad you like it! I did this as a mini-project within our startup MediSearch (https://medisearch.io/) & the search pipeline is custom tuned for the problem.

What are you embedding exactly? Chunks of documents?

Yes

Very promising tool based on a couple of questions I asked it! How did the cleaning of documents look like?

Lots of annoying edge cases as you can imagine, nothing particularly glamorous.

All 10 thumbs up!

Edit: One suggestion: in the results list, please make the headings links to the articles, too.

Done! Let me know if you have other feedback.

Maybe a stupid question, but how do you compare this against GPT-based search engines?

GPT-based search engines usually use some sort of a database to retrieve context for the LLM to summarize first. This is what people refer to as RAG these days: https://blogs.nvidia.com/blog/what-is-retrieval-augmented-ge....

Some of these GPT engines maintain their own vector DB to do semantic search, others are directly hooked into Bing / Google. So pubmedisearch.com would be one component of a GPT-based engine. We actually have a GPT-based engine here: https://medisearch.io/.

What storage did you go for, and what search approach?

We use pinecone and it is not ideal, looking at https://turbopuffer.com/ now. They look quite promising :)

Did you compare pinecone against pgvector with Postgres? Self hosted of course

Isn't it funny how the best Choice somehow always comes back to Postgres in the end XD (for most)

Yes, that’s where I’m these days. I don’t even think of venturing outside of Postgres these days, except for say things like Redis etc. where there are mature and established options for specific use cases.

Will definitely check pgvector, thanks for the pointer.

What kinda dimensions did you keep it relatively low to keep costs down?

Reply on news.ycombinator.com