Challenges with semantic search on transcribed audio files | Hacker News Reader

Challenges with semantic search on transcribed audio files

notjoemartinez.com

I've been trying to solve a problem with implementing semantic search on my YouTube search engine yt-fts (https://github.com/NotJoeMartinez/yt-fts). I've managed to substantially speed up search results by storing subtitle embeddings in Chroma. But a bigger problem has been with how to properly segment the text in a way that accounts for the duration and context of word embeddings while returning precise time stamps. This a blog post exploring what I've tried so far.

Interesting problem and a good write up! This reminds me a little of audio processing where there are 2 representations, time domain and frequency domain. I’m no expert at this but my understanding is if you want to search for “when” some chunk of audio happened, you first need to convert to the frequency domain via Fourier transfer. But then you lose the time info. So you can’t just take the Fourier transform of the whole file, or even 10 second chunks… you have to take a bunch of short overlapping Fourier transforms – overlapping so you get the nearby context, and short so that you have a higher resolution idea of when something occurred.

I wonder if a similar idea would work here, where you could search at various “zoom levels” - first search for an entire video that’s nearby in terms of embedding, then search within 50%-overlapped 60-second chunks, then within 50%-overlapped 1-second chunks.

why not overlap the subtitle excerpts for complete selection options? may be able to have very confident results too. if removing blob conversion speeds it up, is it now fast enough that doubling the table's length is worth it to improve quality? could go for triple overlap?

> […] would have to convert 51,751 embeddings to blob data and then convert them back to arrays when searching which would take over 20 seconds.

0.4ms per embedding is far longer than I’d have expected. I haven’t worked in this space just yet, so I lack knowledge and am curious: how big is each embedding? A few hundred numbers? The bottleneck here is Python, right? How fast is it with Chroma?

—⁂—

Search difficulties in the presence of intermingled metadata and data is not a new problem. I detest issue templates on GitHub because they’re putting everything into one freeform text field, and any keyword that’s part of the template becomes completely unsearchable, because every issue matches the query. This has bitten me more than a few times.

Rather than simply embedding the query, you may want to look into query expansion techniques. You might find that expanding the query semantically and then using more old school (read: extremely fast) tools like bm25 or tf-idf are pretty good already.

Your fundamental problem is that short snippets don’t have a lot of semantic meaning, but long snippets lose specificity. So one way of dealing with that would be to include staggered snippets, like bricks in a wall. This is ultimately a hack though, and for a lot of queries it doesn’t even make sense. If it’s a semantic match and not a direct text match, what does it even mean to give people a short segment?

I contemplated the same problem and concluded the thing to try first would be probabilistic clustering of snippets, but I haven’t actually tried it yet.

My sense is the general version of this problem is not solved in that chunking/splitting algorithms for processing content into vector dbs need to be made significantly more sophisticated and leverage more traditional NLP algorithms.

For example:

https://freeling-user-manual.readthedocs.io/en/v4.2/modules/...

or similar from spaCy or NLTK. and then somehow get to content driven paragraph type splitting.

I've found that 512 characters, more or less, is a good length for embeddings. More than that, and a bunch of models will just ignore the text. Less than half that, there isn't always enough context in the text to be able to group the chunks by keyterm or other labels. I do this type of additive context using the first few results from the embedding search.

If it's tweets, it makes sense to use shorter chunks.

I was dissatisfied with the results of chunks "broken off" from sentences, so I use nltk to find sentence boundaries and then approximate the lengths of chunks. I also built a custom splitter function for Jinja2 that will chunk by length, page offset, and also does overlaps, where bits of the beginning and ends of a chunk will run into the previous and next chunk.

I have a pipeline for this particular type of use, running on an Open Source thing I built to handle indexing texts: https://mitta.ai. The RAG pipeline is under cookbooks.

That makes a lot of sense that a fuzzy 512 character window that preserves sentences properly would be a good strategy.

Do you also use nltk to extract keyterm then? I was looking at Qdrant labels and thinking about how vectors could be labeled with location metadata but also perhaps with named entity information from the chunk.

I imagine it how I might flip though a book using the index and also table of contents and then perhaps skim the intro paragraphs etc. Eventually the RAG systems need to replicate a more sophisticated human like retrieval of relevant information.

I use a dict completion prompt to extract keyterms from various LLMs. Technically, this is transfer learning and I use it in a way that allows set operations to filter the vector space to speed things up. In a version I built of this, I referred to it as a "back of book index".

One issue would be that the prompt for keyterm extraction might miss crucial keyterms, but a subsequent access of those might update them...

Do you have access to the waveform of the video? The most natural segmentation of text here seems to me to be on significant quiet periods in the waveform. An ideal solution is likely adaptive based on the video style which is likely unique to the creator. But I think you could get far with a sound below db threshold for x millis approach.

The transcription/subtitles have punctuation, so I think looking for full stops ("periods") would provide a similar to result to audio analysis. PS. The technical search term for the audio process you have described is VAD.

Reply on news.ycombinator.com