Hacker News Reader

I am working on something similar. I have all the 10-k and 8-k docs.

I’ve pulled out the structured data separately, and now looking at breaking up the text into paragraphs to get embeddings.

Why are you using inverted index style text search instead of embeddings? I can see doing both perhaps…

View on HN

Embeddings did not really work well enough for me, let us say you are looking for numerical text, so for e,g get me AMD's revenue from March 2022. The embedding representation needs to understand that March 2022 together is way more important than March alone, I often ended up with March 2021 or March 2018 as being closer because the text might have multiple terms containing 'revenue' or multiple 'march'... Perhaps I could have improved it, but that did not seem like the right path to go down for accuracy.. This was way worse for e.g when I am looking for ECB statements, they can refer to an older date in their current report and it caused all sorts of trouble :). An initial fix was to basically mention March several times so that the search returns that... :)

Reply on news.ycombinator.com