Hacker News Reader

I have a question for which I haven't found a definitive answer yet: is how can one effectively manage typos and Out-of-Vocabulary (OOV) words in RAG systems?

For instance, if I search for a specific product name but accidentally mistype it, the resulting encoded vector might not be similar to the vector for the correctly spelled product name ?

View on HN

AFAIK, modern tokenizers/vocabs don’t have OOV anymore because they use sub-word tokens if there is no entry for the given word. This also works for languages with composite words.

In your example sum of embeddings for non-typo and typo version should have close cosine distance if it trained correctly.

Good embeddings are supposed to handle this natively. For example the embedding for API should be close to the embedding for Application Programming Interface. Purportedly.

Reply on news.ycombinator.com