Have a look at Opus, another free audio codec that provides lower latency (~22ms) and better quality than Vorbis.

It also scales down to low-bitrate speech, but I doubt you need that for your application.