sanchit-gandhi

A month ago, Meta AI released Wav2Vec-Bert, one of the building blocks of the powerful Seamless Communications models.

The checkpoint is MIT licensed and available in Hugging Face Transformers, where you can fine-tune it to get comparable speech recognition results to Whisper, but with 10x faster inference. You only need 10 hours of audio data, and training can be run on a single Colab GPU!

Hugging Face Whisper (the backend to insanely-fast-whisper) now supports PyTorch SDPA attention with PyTorch>=2.1.1

It's enabled by default with the latest Transformers version, so just make sure you have:

* torch>=2.1.1

* transformers>=4.36.0

Indeed, insanely-fast-whisper supports beam-search with a small code modification to this code snippet: https://huggingface.co/openai/whisper-large-v3

Just call the pipeline with:

result = pipe(sample, generate_kwargs={"num_beams": 5})

There's no need to wait for MusicGen to generate the full audio before you can start listening to the outputs

With streaming, you can play the audio as soon as the first chunk is ready

In practice, this reduces the latency to just 5s