Hacker News Reader

Comment by sriram_malhar | Hacker News Reader

Thank you so much for this course. I plan to go through it properly.

I have a search problem of my own and I have had a hard time applying what I have learnt (including the coursera DL specialization). The chief characteristics are: (a) It is a fuzzy search of a corpus that is in a non-English language. (b) The search should be able to run on a mobile phone _offline_.

Is this possible? Can training be done elsewhere and transferred to TinyML or some such? What would be a good forum to go seeking answers?

View on HN

Have you tried...

a) BM25 after some preprocessing (lemmatization etc.)

b) fastText / GloVe (possibly weighted by BM25)

The results can be surprisingly good. Often no need to bother with big language models or GPUs.

As far as I understand, BM25 is not for fuzzy searches. For a bit more context, the search terms are in the English script, but the words are basically the closest-sounding transcriptions of sounds in various Indian languages. Different people may render the same word differently in an English transcription. But there's enough crowd-sourced data to account for the ways in which words can vary.

For the same reason, GloVe is of not much use to me.

OK, I understand. That is a different problem indeed. Although I am not sure that BERT would be of much more help there either, unless you have quite a large training corpus. The simplest/cheapest approach might be some kind of transcription normalization. As you say, what is introducing ambiguity here is the act of transcribing these sounds into the English alphabet. There’s not a single source language/alphabet? Also, I am not sure if fastText/GloVe really would not pick up the semantic similarity in spite of differing transcriptions as long as you have a large enough training corpus. I’d experiment with the settings here (bag of words/skipgram, min/max lengths etc.).

If the volume of data fits on a mobile phone for it to be offline, perhaps you don't need deep learning?

Perhaps. What alternative would you suggest? The search terms are fuzzy, and there are too many variants (not exactly misspellings) for me to encode them explicitly. So I thought I'd rather learn from a crowd-sourced corpus.

In my case, I don't want the model to be general. I can afford for it to be like a database index, tailored to that data.

Reply on news.ycombinator.com