You are correct. I need submixes for each channel. And, I mis-counted my inputs, and will nix an output, but that still leaves me with a 16x16 mess.

I've spent the weekend looking into this. I would love to build my own, but I'm not sure where to start with the FPGAs. What FPGA would you suggest that I check out for this application? I don't mind buying stuff and experimenting, I just need to know which direction to explore!

Thanks for your comments!

If I can get this to work, I'll definitely open-source my results!