GPT‑Live‑1 in the API | Hacker News Reader

GPT‑Live‑1 in the API

openai.com

54 pointsarittr55 comments

I've been waiting for this! I'm learning Spanish so I built an app to teach me Spanish, but hyperfocused on scenarios in my life, for example "watching a Barça match in a Barcelona bar". It does FSRS flashcard training, and live conversation practice.

I think education is a very underexplored area for these live conversation models. Yes you can just use ChatGPT Live but that's freeform and unstructured, doesn't have a curriculum or can present supporting visuals, etc. On a grand scale if you can give children their own personal individual tutor rather than relying on group teaching alone, there could be a huge jump in successful education outcomes.

Oh yeah, I built so much stuff to learn German, for example [1] to give me random German texts, force me to read it, and answer it, I created [2] to automatically make flashcards for me and then use with with a flashcards app I regularly use and [3] to help me memorise German cases and word-genders. I love it!

I did think of implementing this conversationally, but tbh it has always been too expensive thus far, I gotta retry with GPT-live-1, I tried it with elevenlabs before but it wasn't live enough and the models were not intelligent enough.

[1] https://river.berlin/projects/german-learning-helper/ [2] https://river.berlin/projects/flashcard-generator/ [3] https://river.berlin/projects/german-cases-trainer/

Yeah my current approach till now has been ElevenLabs Scribe v2 transcription, then feed that to Gemini Live. The latency isn't too bad, but it's definitely there.

When you use ChatGPT Live it's instant which is great, although the realtime transcription still kinda sucks, especially if you're a newcomer to the language so you're making mistakes. I'll constantly get responses to something it thinks I said but I didn't say, which is a real hard blocker for a language learning app.

I think what I'll land on is Scribe v2 (the full thing, not realtime) transcribing turns - it is exceptionally accurate for this - and then just feeding that text direct to GPT Live.

Would you say your German has improved as a result? Genuinely curious

Oh yeah definitely, That being said what helped me more than anything is the flashcards app and speaking German with my roommate regularly.

Nonetheless, in complete honestly I do also have a German tutor who I see once a week for 50 minutes, I am very reliable on completing my work though, the "progressbars" in my flashcards app do keep me motivated.

Learning a language is really hard and takes years, but mentally I am convinced, that if the progressbars in the flashcard app I use reach 100% and also in my German cases app, that I will get closer to speaking perfect German, this keeps me motivated.

Oooh thats a good idea and gave me the idea to add a multilanguage flashcard page to my site.

Thanks!

Group teaching for language learning is essential, even more for kids. It’s really important to be in a context where you have to interact with other humans. There is a reason anyone serious and with the means will pay good money to go to language courses IRL to progress, instead of relying on video calls. And the last thing kids need is even less human contact during their education

I'm saying complementary, not replacement. Like at-home homework being a private tutor instead.

I've found the opposite to be true. 1 on 1 with a native tutor is the best. [0]

Group classes are great for language schools (more $) but can be bad for students:

hard to match levels speaking to non native speakers reinforces bad habits

[0]https://en.wikipedia.org/wiki/Bloom%27s_2_sigma_problem

Not entirely joking but group teaching: fire up multiple AIs. Hear live talking in a foreign language, chip in when you want, they adapt to your level. Lack of human contact and alienation are real problems at all ages, but that does not mean current chatbots are not excellent teachers, especially in language learning when you do not even need a curriculum as such, just talk/read/write as much as you can, it's all text.

It’s not really about the number of people, it’s about having a diversity of individuals, interests, topics of discussion, people making different type of mistakes, different accents and pace, etc

Yeah, it's an exciting use case. Although all these models, even seemingly GPT-Live-1 doesn't actually hear your pronunciation, it seems they all get passed transcripts, so for learning to speak another language, they're still not there seemingly.

But, it's close! You can control their pronunciation, make them speak slower/faster, and obviously great at anything text, so many use cases work great for language learning with LLMs. Just wish they solved this last mile thing too!

Some of the models claim to be audio to audio, like one of the Gemini models. But I've tested and it does seem that you're right, it's not getting all the nuance at all

Yeah, sadly "audio to audio" seems to mean "we transcript it automatically for you internally which gets passed to the model", otherwise we'd be seeing models that are able to hear nuance in the input voice and pronunciation, which AFAIK, no model does yet.

the latest Google Translate features based on the all voice real-time 3.5 Live model is as good as I have tried in Live Modes it's not perfect but I can put it down on a table of four or five people conversing and get a reasonable amount of it translated into my earpiece.

Have you been able to find information on which languages are explicitly supported by this API? I'm having a hard time finding that out.

That's great- i recently went down the same path! I was using grok in my car but got frustrated it wouldn't keep track.

Built my own app, currently focused on having it generate stories for me at my level, and also doing FSRS flashcards, using words that i lookup.

What are you using for the voice AI?

Right now I'm using Gemini as it seemed the best value, but I think it could be improved.

Also, how's the learning for you been so far? I've only been at it a few weeks but this method seems to be superior than other things I've tried

Point it to learn catalan! If you want the locals to love you, that's the best way ;)

Yes Catalan is coming after Spanish

Seems like a very popular use for AI, I work on a version for Korean. (A very diffrent feature set, more exercise generation, maybe someday I'll get to live conversation, which would be a great thing.)

However, for voicing sentences I use murf.ai, which seemed very nice for korean.

I tried to talk to the openai version in (my bad) korean, and it responded in japanese :D

Reposting my comment from https://news.ycombinator.com/item?id=49646963

Congrats on the launch here. I've been messing with this over the last few hours- super super cool. I was excitedly awaiting this hitting the API, because ofc there wasn't a super high fidelity option for drop-in voice interface in front of a given harness. This is blowing me away so far!

(Side q, is there a single place one can watch for updates on the API- that actually covers everything that changes? IIRC there have been a couple of additions that you've tweeted- but never hit the API changelog ;] )

And replying with some more thoughts after reading the comments here. To me, this feels similar to when models started passing the line of (imo) "good enough" to start building much more capable agents. The release of this (gpt-live-1) in the app felt like a big jump in capability, and now that this is available on the api- and I've tried it, it really feels like something big is unlocked for devs. Using this as the UI for a harness feels good so far, and its very much just plug and play. I would encourage others to throw their coding agent of choice at the docs- and have it spin up a web page that puts this infront of a capable harness; it feels 1:1 with the current voice mode in the OAI app(s), and you can define the tool surface yourself. Its really cool.

Even as someone really AI-forwards, there are just not enough selling points for me here. I almost never want to talk to an AI. I just don’t believe I’ll have a useful voice interaction. Maybe agents are here to fix that, but theres 30 years of really negative precedent from robot telephone bots to overcome, and I don’t think some new API is going to change that overnight

I talk to AI sometimes to tell me a bedtime story as I'm trying to fall asleep

I can’t tell if you are being serious or sarcastic. Assuming you are being real, and as someone who has to listen to boring youtube videos to help falling asleep, i have a few questions: how long is the story? What happens after the story ends? Like, does it turn off the voice feature or does it stay on and randomly speaks during the night?

hah I just tried it a few times. You have to choose a voice you like (Spruce for me) and then tell it continue a scene or chapter from one of your favorite books

Like for me it's the part in The Wind in the Willows where they're on the cart, before Toad becomes a car fanboy. I asked ChatGPT to stretch that part and make them stay on the road for longer, and it kinda did an OK job tbh

This voice model can do tool calling. So as long you give it powerful tools, I believe it to be useful.

I almost never want to talk to an AI, because text is better for almost anything. But it's nice to have that "almost" corner case covered, no?

Not to mention all the current "telephone bots" applications that could benefit from something that has actual reliable STT and can accurately grasp a number you tell it first try, or hear a natural language description of what you want and immediately bypass listing the entire menu of options one by one.

Can't wait till a customer rep is impossible to get to because all support is outsourced to ai.

I'm not so sure, I think it could go the other way. The vast majority of support cases should be handled in an automated way. I had an issue with Vercel recently where I argued that a bill was incorrect, and the agent produced and offered a refund by itself - that was interesting.

You obviously need humans but they can be freed up to deal with the more complicated cases.

> You obviously need humans

Try to contact Anthropic or Google support, they clearly don’t think so.

Relevant video, just make them say oink: https://youtu.be/fhi6JU5pOJk

I think that this could be useful for the case of learning something by teaching it to someone else, and this someone else being the AI. We all know that learning-by-teaching is a great way to see the gaps in your knowledge and check whether you can explain the topic simple enough for the "student" to understand it. But finding the "student" is the hard thing in this process. Replacing the student with this model, and maybe a better reasoning model behind, sounds like a good enough replacement of a real person, for this case.

Not sure whether the latency between your last sentence and follow-up question from the AI would be small enough for it to not be intrusive. As another user noted in a comment, that it takes time for the conversational model to call a more powerful sub-agent that evaluates your explanation and returns follow-up questions.

Another question is: can the model interrupt you and ask questions right away? What if you're incorrectly defining something and then building up on it? Would the model interrupt you right after the incorrect definition or after you've already finished your explanation?

Been wanting to try this out on top of https://github.com/TristanBrotherton/voicepe-realtime with the HomeAssistant VoicePE

Anyone else hack something together with HA / the Voice PE yet?

Looks like it needs a second model to do function calling, which gpt-realtime-2.5 didn't, and the Voice PE XMOS chip's audio pipeline might not be a great fit for full duplex back and forth, like what gpt-live-1 now supports.

I've given a try to the demo in the webpage. I've asked to tell me which of the first generation pokemon started with the letter C, requesting it to tell me their names in reverse. It got stuck.

I don't know if they are having some troubles with their demo environment due to the volume of requests in this specific moment, but I'm very hesitant to put something like this in production if it fails with this trivial example.

All voices offered sound human. I'd prefer a robotic voice, to avoid over-anthropomorphizing the AI.

The worst use case to show this.

Their use-cases for the recent 'Images' release were also terrible.

I think their 'images' use cases were pretty representative of what normies use it for.

Honestly though they need to hire whoever did those Google ads back in the day.

hopefully this come in openrouter api cause I'm not signing up for a specific provider's specific api platform, and have yet another thing that can bill me.

They wont. This is OpenAI specific. They don't even have support for OpenAI realtime models.

However, this open source project https://github.com/chatbotkit/platform/ does and you can plug OpenRouter or OpenAI keys straight in while keeping your integration work generic. The only downside is hosting it yourself but it is just docker compose up.

Damn, this was a terrible showcase of their voice API, I really love their product. It's the fucking best, a life changer. But... what a meh showcase.

Is somebody from OpenAI hiring? I can show how I use it to learn German, among other very interesting usages.

Also show proper excitement etc... I think also a lot of real users could do better.

It feels like they aren't real users of their own products...

They mention Speak in the blog post but don't show their demo. On Speak's blog post, that demo is way better: https://www.speak.com/blog/live-tutor-lessons-powered-by-ope...

Can it infer someone's accent and correct it or tone of the voice or whether someone is talking in a mocking way? If not, then it's seems not there yet.

> Reasoning & tool calling delegation: GPT‑Live‑1 can delegate reasoning and tool calls to a backend text model like GPT‑6 Astra or a third-party model.

I played around a bit with this in Codex when it became available but even when you have Fast mode + Light reasoning, the mere idea that it passes off actual work to background sessions even for "change this line here" makes it a really frustrating experience.

You can say "Update config here" then wait 2 minutes then finally it comes back, and most of those two minutes was overhead of agent<>sub-agent communication and passing the work, instead of just, you know, do the thing.

I'm eagerly awaiting for this to get ready though, because being able to use tools like Houdini, Unreal Engine and Blender over MCP with this fast voice mode makes for great video game development environment, where you can playtest the game and talk with Codex at the same time, asking it to update stuff on the fly, granted you've setup things correctly.

I used it in a similar way, and once you get it going, it's much faster, since it can reuse the background agent which now has a primed context. I got 30 sec turn around times. And you can talk about the next change while the previous one is being implemented.

> I got 30 sec turn around times.

For a single line change that could be completed in maybe 10 seconds if the agent didn't do the whole communication overhead dance, half a minute is a long time to sit and wait during playtesting to just update some parameter.

> And you can talk about the next change while the previous one is being implemented.

Yeaah, that's not how I work with agents in general, we work on one thing, do it properly and then clean up, then refactor, then testing, more refactor and so on, until a thing is 100% nailed down and properly implemented and then move on. I don't know how people can work on multiple things at the same time, unless they're really simple tasks or small projects. But almost nothing in medium/long-term game development is that simple.

Maybe a one line change takes 2 min because it needs to run the full test suite and so on.

I have special instructions in my AGENTS.md to bypass running the full QA suite for small contained changes, and to run a targeted one instead. And if an error passes through, it will be caught the next time the full suite is run.

I have both a Rust and a Python project, the Python one has a full QA suite, the Rust one is much more bare bones, and surprisingly, I can have Astra implement a small Rust change in 10 seconds, but not a Python one.

> Maybe a one line change takes 2 min because it needs to run the full test suite and so on.

No, it's a runtime parameter, imagine "walk speed", with zero tests as it's a 100% authored experience, tests live elsewhere.

The one line change takes 2 minutes because of the communication overhead, which I clearly stated in my previous comment. One turn having one reasoning block, one tool call and one final reply, is obviously gonna be faster than one turn with a sub-agent which is at least two reasoning blocks, two tool calls, two final replies, two messages passed between the two agents. I'm not sure why this be surprising to anyone that it'd be slower.

You can use regular voice dictation to talk directly to the Astra model, using an app like Handy. Then you have zero message passing overhead.

I do that when I know exactly what I want, and use ChatGPT voice when I need to explore the solution space.

> You can use regular voice dictation to talk directly to the Astra model, using an app like Handy. Then you have zero message passing overhead.

Yeah, but then it's no longer GPT-Live-1 which is what we're discussing here, the point is the bidirectional voice mode and the faster response times...

and yet another showcase of making automated restaurant reservations. It truly is the purpose of AGI, and all software ever, really, to automate that experience.

It baffles me that the labs can't come up with more exciting use cases for voice api.

Client-side reservations are one of those features that looked like the future when Google showed them on Pixel. Years later, I'm still waiting for the iOS ecosystem to catch up

So true. The real use-case outside of the 2-3 specialised domains are yet to come... if they come at all.

Reply on news.ycombinator.com