Yeah the threshold is pretty brutal, but it is enough. Experimentally, I'd say you need under 2-3ms but even at 1ms you can start to hear some phase differences.

Most of the time, I think my synchronization algorithm is actually sub-1ms, but it can be worse depending on unstable network conditions.