I don't think Teensy is a viable platform for this. For digital audio mixer, one of the most important factors is latency and this is where Teensy solution falls apart.

More reasonable approach would be to use something like XMOS chip to handle the streaming and FPGA for signal processing.

That being said, it's still looking like a fun project.