Managing latency is a problem, as you can only encrypt fairly short packets at a time and send each one out, rather than the whole message, and the structure of the data being different can lead to information leakage
I haven't tried, but I'd think encrypting and decrypting a packet would take less than 0.1ms, no?
What's the variation in the structure of the data? Wouldn't you just encrypt and send a fixed-length interval of audio each period?