I have been thinking about doing something similar -- would love to hear more about this if possible!
The software is all custom and written in C, using Jack for low latency audio and my own driver for the display (Adafruit makes a Python driver but it eats half of your CPU and is not optimized for i2c bandwidth or high framerates).