I worked on a project similar when the pandemic started last year. Ran into latency problem and just couldn't find a way around it. Add to the latency issue is also the global metronome as well.