Key to this article is understanding it’s leveraging the newly released Apple MLX, and their code is using these Apple specific optimizations.
Edit: OK I took the bait. I downloaded the 10 minute file he used and ran it on my 4090 with insanely-fast-whisper, which took two commands to install. Using whisper-large-v3 the file is transcribed in less than eight seconds. Fifteen seconds if you include the model loading time before transcription starts (obviously this extra time does not depend on the length of the audio file).
That makes the 4090 somewhere between 6 and 12 times faster than Apple's best. It's also much cheaper than M2 Ultra if you already have a gaming PC to put it in, and still cheaper even if you buy a whole prebuilt PC with it.
This should not be surprising to people, but I see a lot of wishful thinking here from people who own high end Macs and want to believe they are good at everything. Yes, Apple's M-series chips are very impressive and the large RAM is great, but they are not competitive with Nvidia at the high end for ML.
ETA: actually it's unclear from the article if the whisper optimizations were done by apple engineers, but it's definitely an optimized version.
There is also cpp Whisper (https://github.com/ggerganov/whisper.cpp) which seems to have it’s own kind of optimizations for Apple Silicon - I don’t think this was the one used with Nvidia during the test.
I installed following the official docs and found it much, much slower, although I sadly don't have a 4090, instead a 3080 Ti 12GB (just big enough to load the large whisper model into GPU memory).
pipx install insanely-fast-whisper
pipx runpip insanely-fast-whisper install flash-attn --no-build-isolation
insanely-fast-whisper --flash True --file-name ~/Downloads/podcast_1652_was_jetzt_episode_1289963_update_warum_streiken_sie_schon_wieder_herr_zugchef.mp3 --language german --model-name openai/whisper-large-v3
I just ran it again and happened to get an even better time, under 7 seconds without loading and 13.08 seconds including loading. In case anyone is curious about the use of Flash Attention, I tried without it and transcription took under 10 seconds, 15.3 including loading.
Another question that's only slightly related, but while we're here...
Using OAI's paid Whisper API, you can give a text prompt to a) set the tone/style of the transcription and b) teach it technical terms, names etc that it might not be familiar with and should expect in the audio to transcribe.
Am I correct that this isn't possible with any released versions of Whisper, or is there a way to do it on my machine that I'm not aware of?
I was rather hoping for a guide of just how to either adapt classic whisper usage or adapt one of the optimised ones like faster-whisper (which I've just set up in a docker container but that's used up all the time I've got for playing around right now) to take a text prompt with the audio file.
The 4090 is an absolute beast, runs extremely quiet and simply powers through everything. DCS pushes it to the limit, but the resulting experience is simply stunning. Mine's coupled to a 7800x3d which uses hardly any power at all, absolutely love it.
For ex ctranslate optimized whisper “implements a custom runtime that applies many performance optimization techniques such as weights quantization, layers fusion, batch reordering…”
Intuitively, I would agree with your conclusion about Apple’s M-Series being impressive for what they do but not generally competitive with Nvidia in ML.
Objectively however, I don’t see concluding much with what’s on offer here. Once you start changing libraries, kernels, transformer code, etc you end up with an apples to oranges comparison.
That said, for practical purposes, the ready availability of Nvidia-optimized versions of every ML system is a big advantage in itself.
In any event, it’s super cool to see such huge leaps just in the past year on how easy it is to run this stuff locally. Certainly looking very promising.
Also, the price difference between a prebuilt 4090 PC and a M2 Ultra Mac Studio can buy a lot of kilowatt hours.
Wouldn't the high end for Nvidia be their dedicated gear rather than a 4090?
Or having a better image of performance in your head when you buy new hardware.
It's just a blog article. The time and effort to make it and to consume it is not in the range of millions there is no need for 'more'.
Personally, I have already a RTX 3090, so for me it’s interesting if a M3 would be a noticeable upgrade (considering RAM, speed, library support)
The super optimized code for a network like this is one test case. General networks are another.
I mean, you're probably right, but MLX framework was released just a week ago so maybe we don't really know what it's capable of yet.
The power:perf ratios seem about equal although, so it's really up to apple to release equivalently sized silicon on their desktop to really have this be a 1:1 comparison.
Could anyone who understands this hardware better than me chime in on the complexities of bringing unified ram/vram to PC, or reasons it hasn't happened yet?