I have never worked with audio ML but I am sure what you describe can be done.

Latency might be problematic?

Look into projects using torchaudio