Hacker News Reader

Comment by jaggederest | Hacker News Reader

https://taalas.com/ has done it already for a wildly obsolete model. 14000 tokens per second.

https://chatjimmy.ai/ is their interactive. Tiny context, very dumb, but absurdly fast. Imagine this as a tool call for claude code for trivial changes - the tool call from the harness takes longer than the execution.

View on HN

Wow! You weren't kidding,

I just tried it too and 14,098 tokens in .05 seconds, I barely blinked and it was done. There was no typing at all appearing on the screen. It just showed up.

https://chatjimmy.ai/chats/01dc66a4-4b1b-4dea-bb5f-926855e37...

That link isn't bringing up your chat, FYI. It just shows the default new chat state.

Sure enough. I just tested it in incognito, and it is blank. I thought it would work because I wasn't logged in to Jimmy AI. It still shows for me though in my regular session so they must just have a cookie that makes it stick.

Here's a copy and paste prompt if somebody wants to just test it real quick to see what I saw:

Write a story about the fastest monkey who ever lived, his name is Jimmy and he is an AI superbot monkey that is part cyborg primate. He can travel through time and is psychic.

Wow. This is absolutely wild. I didn't expect that.

If we get to anywhere near this speed for the equivalent of the current models... I don't even know what to think about that future.

Speed is the metric I'm currently most interested in. The models are smart enough.

Once speed significantly increases I think we're going to see some interesting downstream effects. The three things I currently spend the most time waiting on are LLM API requests, Rust compile times, and nix derivations. As AI latency approaches zero I think we're going to start taking a hard look at whether slow-compiling languages are adding enough value over Golang, Typescript, or even dynamic languages to be worth the slowdown.

what do you mean "if"? of course we will, and the models will be smarter as well

Yeah. A model smarter than Fable by, say, 50 human IQ point equivalents, running in some kind of autoregressive process continously on whatever goals you put into the loop, on baked silicon, would be maybe my low end for the potential in the relatively near future.

Holy crap, I was not prepared for how fast it responded. I just wrote "Just wanted to see how fast you are! Can you write me a quick story about a tiger who lives inside a block of cheese the size of a house?"

I pressed Enter, and the response was instant.

> Generated in 0.037s • 14,205 tok/s

This is unbelievable.

For what its worth the frontier lab models can surely be a lot faster if they wanted them to be but theyre supply constrained so theyre doing stuff like multi tenancy. Since you cant self host them no one outside the labs really knows speed as a solo tenant

You can kind of get a sense by running these things at home - I'm currently running Laguna. One interesting thing is that per stream doesn't actually slow down that much with multiple concurrents, because the bottleneck remains the memory bandwidth until pretty significant request depths, and then eventually you hit the GPU's limits. It's one of the big forces that pushes for centralization in this stuff, the fixed costs to run one are huge, the marginal costs of additional tenants, relatively small.

It's crazy. Are they doing any precomputing as you type, I wonder if you paste a block of text is it the same speed.

I pasted and instantly hit enter on this prompt: "I generated a filter set using REW v5.31.3 using real-world sweep tone measurements from the room I'm listening in . How can I use it as my MacOS output equalizer so that my spotify music is adjusted for this room and speakers"

and it gave a very reasonable answer in non-perceptible time.

That's a hell of a prompt, you should make a post about what you're doing maybe, because I want that for myself.

idk if you're still actually looking for a solution (I got nerdsniped heh), but I found a seamless way to have it locked into the speakers + room, instead of a specific source/software, is using an iLoud subwoofer that routes to any speaker setup and has room correction + EQ that lives on the sub.

No, it really does take ~0.03s to generate the answer. Try your browser's developer tools and watch the requests.

Isn't it crazy that we can send a message, across the world near instantaneously and have a coherent reply, generated by a computer, sent back to your screen, in under 500ms in most circumstances.

I find myself getting caught up in the sheer speed of modern computing and networking. The fact I can play an online game with 10 other people is just insane.

I got into Rust development via LLM last year, and being able to do things budgeted in nanoseconds is a heady feeling indeed. Real time video and audio analysis? Totally doable, plenty of time budget. 16.6ms is a long time, it turns out.

The magic is in the fact that they essentially have an ASIC llm device. There is no other trickery. The problem they will face is that it is actually locked in silicon, so upgrading models will be difficult, and likely require new hardware each time.

At some point I imagine you’d add a software layer on top that holds more current training and can be called as needed trading off for slower responses. There’s already work out there splitting models across networks. You could have the base on silicon, some stuff in memory on the machine, and another frontier tool in the cloud.

Taalas actually already support LoRA, basically doing exactly what you say.

The other thing that I think is really interesting about all of this, is that LLMs are already perforce behind the times with their knowledge cutoff, so adding an additional ~3 months for bake into silicon isn't such a huge deal, I think, for the ~10x more efficient and faster you get.

Yeah, really a fascinating time in computing. Once these chips start to become more common I'll be curious how people find ways to use them. One can imagine a world where really simple inference is available on dirt cheap chips found in toys and other low cost consumer electronics.

There are already studies proving that a "stupid" model with a good harness + tool calling will outperform a "smart" model.

Things like this give me hope for a system that can be fully local and private, but also with the ability to be almost infinitely extendable with tools.

Can you point me to something to read about this?

Is the size of the ASIC limiting factor or can they infinitely tensor parallelize? If thats possible then it would make it only a matter of economy of scale, there are good enough models already for people to invest in that kind of platform

It’s crazy that you asked the very question whose answer spawned the sub thread, within the sub thread.

I'd like to imagine the things that can be done with this speed and the current frontier models.

Seriously, if Fable or even Opus was this fast that would be a real game changer.

RSI will be models better than Fable running faster than this, you won't even need a human in the loop to figure out what to do. The high level goal will be accomplished better than the human in an instant

Truth, it feels like we're in the dial-up age of LLMs right now. And this Jimmy AI is fiber.

Fully interactive games where you can talk to every NPC by text or voice and have an LLM drive the story (with your own meta prompts to guide it, if you so wish). Maybe even have them generate assets on the fly too.

I’m still trying to figure out coding agents. I can’t even begin to imagine the things it would enable. Even the most mundane ideas like LLMs-in-HiFreq-trading have huge implications.

0.03 seconds is an eternity in high frequency trading

You need to be 4 orders of magnitude faster at least

Just wow, I made a similar request. Result: Generated in 0.042s • 14,201 tok/s

This is crazy.

"Stochastic gradient descent algorithm in Haskel"

"LMS algorithm in bash"

Just barfed it up lol.

Amazing.

See Cerebras and Groq as well.

Wow! Responded essentially instantaneously to my prompt:

"I need a short, 4000 word essay on the the difference between star wars and Star Trek universes from the perspective of graduate level scientific work."

Answers like 8Bish model it appears, but instant response

hugged?

Reply on news.ycombinator.com