The most important thing when you do this is that you don't listen to your local audio signal, you only listen to the mixed signal coming from the server. That puts you in time with everyone else, more or less.
Roundtrip latency was under 30ms, which you definitely feel but your brain can compensate for.
The funny effect overall though is that because you're delayed and everyone else is delayed, you end up sort of waiting for the other player, and they end up waiting for you, so instead of the classic band "everyone speeds up" it's like everyone progressively slows down as the song progresses.
We haven't tried with our drummer actually playing drums, he's just been recording against other parts we provide. I don't think it'll work very well because his acoustic kit will be too loud and he'll hear that as well as the delayed drum sound, which for him will feel like a stuttery mess