Is anyone working on audio libraries that will enable streaming audio chunks for Whisper processing? Saving audio files into a local file system, running ffmpeg to chunk, and then sending them off to Whisper is very tactical..
On the other hand, if you expect a variety of formats (mp3, wav, etc), then shelling out or embedding ffmpeg is probably the quickest path to achieve something. The Membrane Framework[1] is an option here too which includes streaming. I believe Lars is going to do a cool demo with Membrane and ML at ElixirConf EU next week.
[0]: https://github.com/elixir-nx/bumblebee/blob/main/examples/ph...
Yes, the relevant part of his demo with the membrane pipeline appears to be here: https://github.com/lawik/lively/blob/master/lib/lively/media...
I have a rough one using Membrane (media framework) that you can find here: https://github.com/lawik/membrane_transcription
I am using it for this talk I am putting together for ElixirConf EU so if you want it used in context that might be helpful: https://github.com/lawik/lively
Neither is release-worthy levels of polish but if interest is there I should make a proper library out of it.
That is to say streaming chunks works great already. I would love two things. Stitching the edges of the chunks, would probably need to do overlapping for that. And building chunks based on silence. That's more DSP than I know though.
I think this work is very important. I don't understand whether I actually needed to install the library dependencies for Membrane's sake or specifically for this use case (mad, ffmpeg, portaudio). Doesn't feel right..
I'm not an expert here, but I'd expect that capturing a sample using Membrane and piping it into Whisper should be doable.