I would start with rpi-buildroot and build up from there. If you have low latency audio questions feel free to email me.

https://github.com/gamaral/rpi-buildroot