Lessons learnt building a real-time audio application in Python
vangemert.dev
vangemert.dev
"Latency seems to shift by a few tens of milliseconds when restarting the application." - this makes me think you are using the wrong API for your sound input/output. With modern realtime audio support, your total latency from input to output should be less than 10ms total.
"I expected that memory usage would get out of hand quite fast due to the ever growing dictionary of arrays containing audio data, but this does not happen in practice. I suspect that the good performance is caused by highly optimized memory management of Python and modern OSes." - without concrete figures it's quite hard to evaluate this, but what did you expect to happen? With a 44.1KHz stereo audio stream, you should be storing 88.2 thousand samples a second. Say you're using 64-bit floats, as a worst case. Your audio storage should be growing at about 689KB/sec, plus a bit extra for object overhead. How much is it actually growing by? Of course Python is probably doing a bunch of allocation and deallocation for temporary objects behind the scenes, but hopefully you should not need to lean too hard on 'highly optimized memory management' - ideally, you should hardly be allocating anything at all. Also, why a dict, rather than just a large array that you can occasionally make bigger?
Finally ... I'm sure you already know that Python is possibly the worst mainstream language you could pick for realtime audio processing. But that is fine. I have tried to build audio stuff in Python too! Sometimes using the wrong tool for the job is part of the fun.
Just to be clear that you're measuring apples to apples with OP:
You are measuring less than 40ms roundtrip latency on your Mac. Is this correct?
The reasoning behind my comment about round-trip time was as follows:
- Right now I'm measuring around 300 ms round-trip time, without processing inbetween
- In the past I've tried to do live effects in Ableton with ASIO drivers (guitar in -> Ableton effects -> out), and the delay was too noticable. I couldn't play that way without making my ears bleed and I've switched back to pedals since.
10ms latency is how long sound takes to travel 3-and-a-bit metres. So if your amp is a few metres from you, you would experience that delay between hitting the guitar strings and hearing the amplified sound. This should barely be noticeable. If you were noticing a delay greater than that in your Ableton effects setup, your settings needed tweaked. All of this is completely possible - I had a PC-based electronic drum setup in 2006, running through the Reason DAW, which had 8ms latency between hitting a pad and hearing the result.
Hmm, I wonder if Cython (static Python-to-C compiler) would make writing audio code easier/more possible?
I'm a bit on the fence about what this means for the difficult latency calibration routine in the application. Ideally I could throw the calibration routine away, but then I require that users have ASIO installed, while the app now also works with non-ASIO drivers. And indeed Python itself might become a bottleneck (making this work in Python has been half the fun).
About another interface: I do want to keep the application supporting cheaper interfaces such as the Scarlett, because the target audience (hobby musicians) will be using those. Still would be a nice upgrade for me!
If you get 300 ms you're doing something wrong. Note that Windows has multiple audio APIs, 300 ms is about the latency of the old MME api, you need to use the newer one, WASAPI.
Put another way: any Windows audio device supports WASAPI unless it only ships with an ASIO driver which is unlikely, even in the pro audio space.
Crazy numbers on top of a dictionary of arrays? It's all there.
I must defend the design choice for the dictionary of arrays though, this has been a very conscious choice:
- The "dictionary-of-arrays" approach allows lookups in constant time O(1), irrespectively of how much data has already been stored (compared to one big array)
- The dictionary structure allows me to throw away data in the middle easily (without having to handle growing arrays), because the "dictionary-of-arrays" has already been chunked. The audio looper will use only some parts of the recorded audio, leaving big parts in between unused.At least BASIC was designed for native code compilation from day one, and after the 8 bit home computers generation passed by, getting compilers for 16 bit home computers was rather easy.
30 years later, people insist in using bytecode interpreted language for the wrong use cases.
There are enough alternatives with the same dynamism and native code generation.
But as others have said, it’s not a great tool for real time audio since that overhead starts to be a larger % of the time spent. You might want to try compiling parts of it with Cython and using the C API of numpy with that (you can do cimport numpy in your Python code and it’ll remove much of the overhead as it’ll call the C functions directly rather than their wrappers).