Lots of recent development in the speech-enabled LM space recently (see https://github.com/ictnlp/LLaMA-Omni, https://github.com/gpt-omni/mini-omni)