Yes. Have you cheched livekit? Vapi is also built on media framework by daily.co. They have an open source version of voice ai called pipecat. You can check it over github.
Piecat is good project i had used for building voice agents but not enterprise ready need to do lot of work to deploy.