I don't know much about audio encoding, but do the speakers not have to buffer the incoming packets? Large enough buffer size would introduce drift between speakers even if everything is fine network-wise.
You only care about delay between the speakers, not about what latency any speaker has relative to the source.
NTP is accurate enough for this, but I think most of the modern protocols in the wild e.g. AES67, AirPlay2 are using PTP. It is both more accurate and in some ways simpler for this use case.