Hugging Face Whisper (the backend to insanely-fast-whisper) now supports PyTorch SDPA attention with PyTorch>=2.1.1
It's enabled by default with the latest Transformers version, so just make sure you have:
* torch>=2.1.1
* transformers>=4.36.0
It's enabled by default with the latest Transformers version, so just make sure you have:
* torch>=2.1.1
* transformers>=4.36.0
I just reran the notebook with 4.36.1 (minus the to_bettertransformer line) but it was slower (the batch size 24 section took 8 vs 5 min). Is there something I need to change? Going back to 4.35.2 gives the old numbers so the T4 instance seems fine.