Yes, that sounds right. Whether this sort of approach works depends a lot on whether your instrument is naturally quiet enough that you can just focus on what you hear from the server. People are actually surprisingly good at adjusting to instruments that are "slow to sound"; just like you can bounce a basketball on the beat with a little practice, even though it takes many milliseconds for the ball to hit the ground after leaving your hand, you can learn to play keyboard etc with a similar delay.