Hacker News Reader

Comment by jjcm | Hacker News Reader

I'm still sad that we haven't seen a new Taalas style chip a la https://chatjimmy.ai/. Smaller models are good enough now to make that insane burst of tokens so useful.

View on HN

I don't know the model behind this, but it is absurdly bad.

> Write me a coherent paragraph in French, without ever using the letter "e".

> Voilà une phrase claire et concise : "Le village est situé dans les montagnes. Le soleil est haut. Il y a des animaux dans le village. Il pleut dans les montagnes."

I suppose this is just a demo of how fast an LLM can be, I wonder if there are tradeoffs with larger/smarter models. Also, for a human usage, at what point are tokens generated fast enough that it's pretty much instant? My bet is below 1000 tps

Its Llama 3.1 8B, a very old/small model.

for comparison, Qwen3.8-Flash-Next only requires 6B parameters for computation, but stores 125B, 51B of those can be comfortably offloaded as they're not actively used in decode but a single token look up.

The Quant iQ4 of this model loads, then, in ~60GB of vram, and on disk it's 85GB.

So if you could etch it, you'd need a ~25GB ssd chip and 60GB of vram.

The vram costs likely contributed to these things being out of reach of the current economic cycle.

You don't want to use a sparse model for a Taalas-like design. Something like a Qwen 3.8 27B makes much more sense.

I'm not smart enough to know why; I do know that 27B is greater for short/interactive on blackwell, but the intellgence leap of the MoE in Qwen3.8-Flash-Next is quite remarkable.

I'm pretty convinced the pathway to local models will be MoE, especially if they can find a way to keep tweasing out things like PLE into the slow bandwidth lanes.

The Taalas architecture makes loading weights free, but they basically has to pay the same silicon for every weight, whether it's used or not. MoE models are more efficient than dense models per weight you load, but less efficient per weight you have to store.

MoE models are the path to local models with traditional system architectures, but they are antithetical to what Taalas was doing. If you spent all the money to etch 125B weights into silicon, you'd want to activate them all for each token, instead of only touching 6B. You cannot match the 125B sparse model with a 27B dense one, but you might be able to match it with a 60B or so one.

Perhaps, but every time i conceptualize dense models, it seems like its overfitting, and the prime attention is never going to be that dense.

Wasn't it also quantized aggressively, like 1 or 2 bits?

That's the problem with etching a model onto a chip: by the time you've designed the chip, manufactured it, tested it, shipped it, and deployed it, the model will be hopelessly outdated (with the current improvement rates). And when you want to update, you have to buy new chips instead of just uploading a new model file like now. When Taalas announced their chip, the model was already 1.5 years old (stone age by current standards). It's their first chip, so maybe they can streamline it, but the problem of having to update hardware every few months to keep up with the industry is not going anywhere.

You can upload different weights and even do LoRAs. The chip architecture is interesting, the first (n) layers are the sane, so you can change architecture by adding (m) layers. Plausible that this is sufficiently flexible enough for several generations of real world applications. For example, we still use 45nm general purpose silicon for automotive, e.g.

What they did had never been done before. Now we see that it's possible, there are plenty of models to choose from that could be etched into silicon. In the next year or two, I think these smaller models might plateau, and there may be some on-device niche they can fill.

Then again, good luck writing a coherent paragraph in French without an "e". :-)

There is a book written under this premise. Probably the inspiration for that prompt

https://en.wikipedia.org/wiki/A_Void

Why ask this when we know that LLMs are not good at the character level. They run on tokens, not characters. In fact, they don't even see the characters, unless you do special tricks.

I asked it to translate your sentence to English and it did fine. In less than a fraction of a second.

To be fair, you picked a well-known tricky benchmark for LLMs: When working on an embedding spelling disappears after the embedding level. I imagine modern frontier models have tools that let them read back their input to work around this issue.

Pretty sure context is in SRAM, and then you have that as a blocker for tasks.

way less of a blocker these days due to sparse attention...

Reply on news.ycombinator.com