Interaction Models
thinkingmachines.ai
thinkingmachines.ai
Speaking of taking my money, what's the economic model for a company like this? They've published a fair amount about their architecture - enough that I imagine frontier labs could implement. Patents? Trade secrets? It's hard for me to understand how you'd be able to beat that training compute and knowhow at Anthropic/GOOG/oAI/Meta without some sort of legal protection.
I can't wait to see what these model architectures do with like 30-40% lower latency and more model intelligence. Very appealing. For reference, these look to be roughly 1/10 the size of Opus 4.7 / GPT 5.x series -- 275B, 12B active. So there's lots of room to add intelligence, and lots of hope that we could see lower latency.
i think the real ones know this is the tip of the iceberg? hparam tuning, data recipes, data collection, custom kernels, rl/eval infra, all immensely deep topics that would condense multiple decades of phd lifetimes to produce SOTA performance (in both senses of the word) like this.
i would also calibrate what you are impressed by. simply waiting is a posttrain thing - the fact that gemini and oai have not prioritized it is not something you should overindex on as hard. what they showed with full duplex is technically far far harder to achieve
I’d like to believe from the demos that this ability to wait kind of falls out of the model as an emergent property — perhaps coming out of a small RL loop - rather than a specific behavior trained, a-la a VAD component in a stack. Either way, I would guess that VAD absolutely cannot do this right now — interruptions are highly annoying on all voice interaction experiences, and if it were a simple matter of better post training, SOMEONE would have done this, e.g. elevenlabs.
But, I disagree on your idea that this is too expensive/too hard to replicate. For me, yes. But, there’s an existence proof — a small team at a new company just did this without a real roadmap, certainly for less than $1b dollars and probably in less than two years. They are almost certainly less skilled at your list of needs to replicate than teams at the frontier labs, who have been given a roadmap.. So I don’t think it’s as difficult as you propose, from an organizational skills perspective.
This is ML/AI. Is not calling third party APIs. If you want any SOTA in any AI area you need to design your own strategy and models. Drilling down to get there is super painful and perhaps not something a paid-for-course can teach you.
Random is everywhere and so are unexpected engineering challenges. Mastering linear algebra alongside some geometry and still knowing classic algos is the starting point.
Let's say a cutting-edge young researcher is making a name for themselves in their field and earning $300k/yr at a company where they're encouraged to publish and speak. You're trying to headhunt them for a company where they'll be forbidden from sharing their work which will likely stall their career and reputation outside of that company. How much do you think you'd have to offer? $600k? $1M? $1.5M?
When faced with the choice to paying significant salaries, hiring lower-tier researchers, or just letting their people publish, many companies conclude that giving away some of their work is the best option. (And that doesn't even include the benefits of boosting the company's profile which makes it easier to attract other cutting-edge researchers.)
oh, honey.
Edit: when I edit this comment they have been merged in the form so I speculate this is an iOS keyboard feature.
tinker - for fine tuning a custom enterprise model,
interaction models - for working as a digital paired employee (as opposed to a company having to reinvent their entire process around ai agents)
> Time-Aligned Micro-Turns. The interaction model works with micro-turns continuously interleaving the processing of 200ms worth of input and generation of 200ms worth of output. Rather than consuming a complete user-turn and generating a complete response, both input and output tokens are treated as streams. Working with 200ms chunks of these streams enables near real-time concurrency of multiple input and output modalities.
That's probably the main thing that distinguishes it from the multimodal models from other frontier labs as far as I can tell.
We can do these things today, but they're "bolted on" as afterthoughts. Yet they work remarkably well. I wonder how well they'd work if trained int his combined regime, from the ground up.
How does this work? Don't LLMs/transformers need whole context to output next chunk of tokens?
But I completely cracked up at the unexpected physical comedy of the woman in the "slouching" demo, haha omg that was comedy gold, no notes...
I do appreciate less of that flavor of demo that we get from OpenAI/Anthropic, and more of this "human"-feeling vibe. Dare I go as far as calling this an example of "human-centered design" even (https://en.wikipedia.org/wiki/Human-centered_design)?
Every demo by openai showing of their models is "tell me how tall the statue of liberty is divided by the year the inventor of steam engines was born". It's cool but it's so hard to find an actual use. As a personal answer machine I find it very useful but if someone told me 5 years ago; here's a natural language computer as smart as at least every 15 year old, it costs a few bucks per million words. I would have thought that the applications would just scream out but till this day - outside of programming (a big deal tbc) - no one has found a good use for intelligence. It's so so weird.
I guess even a company can't just automatically make more money by hiring more people but I'm still confused
Presumably it will be possible to adjust that behavior with settings, the system prompt, etc. Not that most users will make such adjustments, though.
I'm currently teaching a class on AI-related issues at a university in Tokyo. Many of the students were surprised when I showed them that they can change the response behavior of chatbots to make them more or less verbose, sycophantic, etc. It shifted the direction of our discussions on the possible impacts of AI on the people who use it.
It seems that in a lot of cases you would be able to iterate faster on the user interaction harness if it's an external harness rather than a full-blown model. For example, if there's a UI standing between the user and the model that needs to change (perhaps by the user customizing it themselves).
IMO flexibility is mandatory because for fixed use cases like live translation or a straight-up voice bot, sure a model like this helps, but in each of those cases you'd just be outcompeted by even more specialized alternatives down the line.
With the exception of the real-time translation (which seems like it should be a separate product all by itself), none of the use-cases they presented had much utility. I don't want anything to count the number animals in my stories or time a trivia quiz for me. The auto-slouch-detector, while the demo was pretty funny, just seems so dystopian and weird. AI interrupting you to scold you about taking elderly parents mountain biking instead of waiting for you to finish to scold you? No thanks.
The UX is also an issue - the model interrupting the user (even when apparently required by these strange use-cases) is jarring and makes one lose their flow. You can even see this in the demo videos that they put out - the employees/actors had to really concentrate to continue speaking as if they weren't being interrupted by a brash robotic machine. A human, when participating in this (rare) "invited interruption" has the ability to speak "under" the main speaker and I feel it's generally timed with a lot of nuance.
Even in the auto-translation demo, they ducked the human's audio but the AI steamrolled him and it would have been impossible to actually do that demo without either an incredible amount of control over one's speaking, or (more likely) muting the output. A human translator has a way of "pointing" the "output" to the intended speaker.
The very best part of this tech was presented in the first video where it shows the AI not needlessly interrupting the user. This seems to me more of an important bug fixed that the current models still (somehow) have.
Maybe a good use-case for this would be counting "um's" and the like while practising public speaking.
- Voice assistants
- Customer experience
- Gaming
- Meeting assistants
- Real-time coach or user assistant for using software
- Translation
- Real-time work on a computer controlled by voice (frontend / mobile dev, CAD, 3D modeling, etc)
Traditionally a lot of these use cases with LLM agents are higher latency because the model needs to wait for the speaker to finish, then decide to call a tool or respond - if they call a tool they need to process the tool result and decide if they want to call a tool or respond, etc...
Local models will catch up soon.
is it separate batches of special "skills" that are added post training? how can they guarantee the models won't eventually lose a skill?
Next gemini releases coming next week though, we will see how that matches up!