Hacker News Reader

The rather complicated prompt to force the recipes to be in JSON w/ constraints can be substantially simplified using ChatGPT's function calling/structured data capabilities: https://news.ycombinator.com/item?id=38782678

Additionally, a full vector store is overkill for a menu with <100 items. You can just keep a numpy array in memory with vectors for that and use a matmul for RAG.

View on HN

> You can just keep a numpy array in memory with vectors for that and use a matmul for RAG.

... what? Tell me more.

The model they use (BAAI/bge-small-en-v1.5) produces embeddings 384 wide, at float32 that equals 1536 bytes each. The size of all the vectors for 100 items is 153kb. Calculating the dot product of that against a query will be under measured in nano seconds, even with a naive implementation.

This is actually what llama-index does if you don't use a vectordb integration

How else would they sell managed vector dbs and llm app frameworks? /s

Reply on news.ycombinator.com