whisper.cpp is an open source project that has a feature to turn whisper into a real time transcription program, it works surprisingly well by feeding small segments of audio into whisper. take a look at their GitHub page!
You're right, and it also is apparently optimised for m1.