That looks like a lot of fun... can you share a little about what you used to make phones work nicely as buzzers?
Obviously latency can be an issue, so my non-perfect B+ solution is to essentially calculate the round trip of the buzz, divide it by 2, and subtract it from the server time.
client ------------ pingTime ------------> server
client <----------- serverTime ----------- server
roundTrip = pingTime + pongTime
buzzTime = serverTime - (roundTrip / 2);
The server time is always the source of truth since client times can vary. Not using NTP which could possibly be more accurate.