So long as the amount of audio sent in a packet is larger than how long it takes the next packet to get there, you'll be transfering audio data faster than it is playing, making minor fluctuations in timing between packets irrelevant.
Bluetooth isn't used because BT audio transmission is lossy, audio gets recompressed. Newer BT standards have good quality compression, but you are still recompressing the audio. BT isn't meant to be a high bandwidth protocol.
BT's latency around audio is because 99% of BT implementations suck. Latency can be as low as 50ms or so, but it is often in the 100s or even 200s of ms.