Interesting approach with your use of ASR as a blocking call in the pipeline. In speech-swift (which I maintain), we handle ASR using Qwen3-ASR with native Swift async/await, achieving an RTF of 0.06 on Apple Silicon. This might offer a different perspective on handling latency.
https://github.com/soniqo/speech-swift