> I have increased my latency tolerance to about 100ms now.

Every musical instrumented (voice too) has a latency. The practiced musician is trained on the latency of themselves and their instrument and will initiate a musical note at the right time. That might even be a few seconds in advance.

Part of learning an instrument is learning its latency. Jamulus adds a bit on top of that. That requires some getting used to. A good tip is to listen to your sound over the internet. So wear insulating headphones that keep out your real-time sound.