Ok, real life example: I now spend most of my time, as a developer, waiting for the agent to do its thing (after careful prompting, I'm also thinking about work stuff, don't worry I'm not useless). What if it gave back the same excellent results, but instantaneously? Why, then, I certainly would become the bottleneck. So, quite possibly, my last work task would be to plug this agent directly into the ticket system where the domain experts input their feature requests. Maybe we still need 1 developer out of 100, to coordinate releases and all that (ok, say 1 out of 10).
But that's not taking things far enough: why do we need these domain experts at all? Our pitch is clear, and all software-enabled, though it took years to develop. We can just have the clients express their concerns to the AI, directly or indirectly. Have multiple lighting-fast agents with different roles (refactoring agent, new features agent, debugger agent, domain expert agent, etc.). So we fire everyone, maybe keep 1 product owner / devops to keep the trolls out. The cost is still probably 100 times less than it used to be (beyond the initial cost of acquisition of the magic machine or whatever).
But one of these clients, surely, will realize that these 10 years of manual and slowly-automated development can now be emulated in very, very little time. Why not just, say, take screenshots of the entire app and feed them into the magic machine? Why, this way, they could have the service for a tenth of the yearly cost, forever!
And then the economy implodes.
I'm not saying it's THE most likely version of things, I'm saying that at a certain level, quantity (or rather, speed) is a quality all its own. And this new quality might change the world. Let's hope it's for the better!
It would certainly be an accelerator for people who know exactly what they want. And it would remove multi tasking, which I‘d appreciate.
This is the "dumber but honest person that works harder" phenomenon, vs "lazy genius".
Sure, in the future full rewrites and stuff like that will be just another "throw money at it" problem, but fundamentally software can get arbitrary complex and we barely know how to write large, maintainable code bases.
Nonetheless, I think testing (and maybe proofs) will have its long-awaited time to shine, as being the "reward function".
Right now, I put models in low thinking mode during my refactors and hate waiting. I would much rather have a faster model that that maybe was slightly stupider, and I would wait far less long between prompts where it needs my valuable input.
Models that are dumb, but humble and fast, can be fine.
Obviously a CTO is not going to walk away from the technology just because it's not good enough. That much more incentive for someone to create a powerful enough harness that can direct that power safely and productively. Like a nuclear core, we'll need to come up with the graphite rods and water tank. And if tokens are essentially free, why not, for every million tokens, spend 10x tokens on code review, testing, etc?
Whatever you can cheaply do with AI is not a moat, if there is profit in there there will be quick imitation and competition will eat away those profits.
Models can be replaced easily, harnesses & AI tools too. And if cloud inference gets too expensive there are local models keeping the cloud prices hard capped.
Probably AI won't make anyone very rich.
If inference speed goes up, I can launch the same query 5 times, evaluate the best result and proceed from there. Of course, evaluation is also instant, so in seconds I can get a near perfect solution. Or maybe 10 and I can pick what I like the best.
AI previously provided speed but not quality. As soon as quality reached an acceptable threshold, the speed became the reigning factor.
In my opinion the quality is still much lower, but speed means the cost is significantly lower also.
I mean, if an agent can do half-decent work in less time than it takes the user to prompt them (and "user" in this context is a fast touch-typist like most programmers are), it's obvious it's not the agent that's the bottleneck anymore.
This has always been the case for human project management, and LLMs just aren't at that level yet.
It's more like everyone is speed running to how fast they can convince others that "half decent" is good enough. And for sure, newer models of LLM seem to be getting better at that.
But that's what Agile is all about, isn't it? We've been speedrunning delivering increasingly smelly shit at increased velocity ever since SaaS became a thing, because ubiquitous Internet access is what allowed our industry to adopt the "lob feces over the fence for users to deal with" release model.
AI does speed that up, true (though since the market - and management - didn't catch up with it yet, we have a brief moment where we can use AI to increase quality while keeping usual delivery rate.)
That said, it obviously depends on the project.
A frustrating vision of the future would be when we've been asking for faster loading lighter web pages for years and then companies start caring about it and improving it not for us humans but for LLMs.
you need to launch 10-15 more terminals, who is waiting these days? :)
This model had zero information right, while being fast in responding.
Unacceptable.
> Bruce Lee was born in San Francisco, California, USA on November 27, 1940.
> Bruce Lee's father was a Chinese opera singer
That being said, this is not a good test. It is a language model (a very small one), not an encyclopedia.
ChatJimmy interface is just a tech demo. Without tool calling functionality we can't expect it to be factually correct.
This will generally make them suck, though, a little bit of randomness is necessary for proper function.
I just pasted your comment and its whole inheritance chain to it, started my comment, and asked to generate a total of 9 completions, 3 from each of {current & next word, current paragraph, current paragraph + rewrite the entire paragraph}.
Half of the answers were perfectly good (ironically, not the "next word" ones!), but the important bit, they came back near-instantly ("Generated in 0.024s - 14,163 tok/s", the page says). Slightly more powerful model while keeping this under a second, and this could easily become a qualitatively different form of autocomplete/text suggestion. Running in the background every couple keystrokes, or every time user stops typing for more than 500ms.
>I just pasted your comment and its whole inheritance chain to it,
Good idea. Only problem is it doesn't work. I just did the same thing with exactly this prompt:
>did the user IOT_Apprentice participate in the thread below and if, number and quote all of their comments. Only just number and quote the comments or write "Did not participate", do not add any commentary. Quote any comments by this user verbatim, exactly as input. Thread:
followed by pasting the thread[1]
And received the answer "IOT_Apprentice did not participate in the thread."[2] in 0.001s, even though they have literally the last comment in my quote and it's clearly legible.
It's particularly insidious because the understanding and thinking that is required to follow my requested answer format exactly is substantial - so based on the fact that it gets the format right and clearly understood the assignment, I would be inclined to believe that it would also be correct!
So to use your example, it's not just autocomplete, it's autocomplete that confidently returns "No matching results" in 0.001 seconds, even though there is a search term matching what you put in, right in the prompt itself that was sent to it. That is much worse than useless.
[1] prompt: https://ibb.co/CKVmRvtd
[2] result: https://ibb.co/BKdRKmyD
Autocorrect that works. Reply suggestions that almost work, just need to be tad more accurate (probably more of a data access issue than model) and a tad faster to look completely seamless. Screenshots with automated text detection and OCR and automatic interpretation (different suggested actions for when something on the picture looks like a web link, phone number, postal address, e-mail, or QR code, or an event poster). That's just a fraction of things I saw showing up on my Samsung phone over the last 6 months.
For over a year now, you could get a much better autocorrect and spell/grammar check, and a translator all in one, if you just pasted your text to a frontier model and asked it to check for errors or translate into target language. Now imagine being able to go through a round of such checks in a 1/100 of a second. You could have this running every keystroke, and suddenly the inline autocorrect/checks would not suck anymore.
Auto-linkifying that can correct for typos and doesn't need careful regex tuning because it understands from context what is meant to be a link or not. That's just one of many obvious things possible once you get local models running fast enough. Tip of an iceberg, and the first step to imagining all the other potential uses is to let go of the two mistaken beliefs people hold on to:
1. That LLMs are about written language. They're not; ever since "multimodal models" became a thing, tokenization extended to visual and audio space, and now textual and visual and aural inputs are all just regular, first-class tokens.
2. That chatting with the models is the only optimal way for end users to interact with AI. That's just artificially limiting yourself to the space of chat-based UI.
Raw intelligence becomes slightly less important when you can iterate and improve automatically. You can still claim it was "one shot" even when 30 different implementations were made then combined.
But if the basic premise of "good enough LLM at insane throughput" holds, I think it could qualitatively change local uses of LLMs. At a certain speed point, you're able to move from request -> response to a cascade of tool calling and "subagents", which could allow a small model to be much more useful, if provided with a lot of local data and tool calls.
That said, this is assuming you could stuff a "good enough" model into a phone with Taalas-like technology. The Taalas tech demo was an 8B parameter model and required hundreds of watts (IIRC) to run. The efficiency was good given the speed (as I understand), but it's not clear at all that the approach scales small enough to be a sensible coprocessor on an iPhone or whatever.
That also needs server class hardware though. A phone won’t happily service the insane amount of IO, compute, and network that this cascade would require.
Fully interactive realtime NPCs in videogames at scale.
Recommender systems that simulate individual consumers.
Crazy shit
In a way... when it's finance, they should be maybe called Gray'ish Swans?
Better yet use it to dimulate counterfactual phenomena like market manipulations ypu intend to enact...
I could see it being feasible to get a Qwen-3.6-27b type of model done on something like this. Qwen-3.6-27b at 18tok/s would be a game changer.
8B model (FP4) = 4 GB DRAM = 32 Gb DRAM = 80 mm2
8B model (Taalas) = 4 GB ROM = ~800 mm2