The Unofficial Guide to OpenAI Realtime WebRTC API

webrtchacks.com

53 points·feross··3 comments

If anyone has questions about WebRTC + Realtime API I would love to answer them. I am incredibly/excited about the IoT/Embedded side of it especially :)

You can see a video demo of it here[0]. The code on microcontrollers is IMO delightfully easy to get going on [1]

[0] https://youtu.be/14leJ1fg4Pw?t=824

[1] https://github.com/espressif/esp-webrtc-solution

Looking forward to stable release. We had a few times when the model hallucinations were very funny. In two cases one a call to a roofing company and in another a call to a search ads company the model decided it was a young girl sad about getting accepted into music school and then also super excited about… was very random. Hoping the stable release addresses some of this maybe giving more control over the temperature… also we noticed if you pass an empty tools array the api silently crashes … I realize these are maybe not super WebRTC related but limited the ability to scale out this as a solution to more people. Also would be super cool to just allow direct sip connections - would allow bypassing the livekit requirements [edit ] - ah you are livekit ! We love livekit ! I thought you were with OpenAI - maybe they will release stable soon and address some of these issues but really just wish they had a direct sip way of connecting that would be incredible

> also we noticed if you pass an empty tools array the api silently crashes

I will go look into that!

> Also would be super cool to just allow direct sip connections

Soon hopefully! We have all our own SIP infrastructure (from 1800-ChatGPT). Just need to expose it as an API.

People much smarter then me are working on the hallucinations. It's a pretty interesting world writing evals and improving the models.

Reply on news.ycombinator.com