Hacker News Reader

Comment by dd8601fn | Hacker News Reader

Is this the chatjimmy asic approach with a bigger model?

No, the asic could only ever run one model/set of weights, no updates possible, ever. These are general purpose processors that can have their models updated. But the chips are enormous, with a substantial amount of on-die memory alongside the execution units, for a relatively insane amount of memory bandwidth.

I thought from what I read about the Taalas approach, the model architecture and overall size couldn't be changed, but model weight values could be updated after for further tuning.

Not as flexible as Cerebras though. And I'd love for someone who knows more to clue me in to the truth.

Nah, Taalas was putting the weights into silicon as a mask ROM. Their demo chip was hardwired to serve Llama 3.1 8B, and could never be updated. New models, even new versions without any architectural/size changes meant new tape outs.

But in exchange, you get insane speed and great energy efficiency. I could see it being a great approach for basic "good enough" models.

They may have had a little flexibility by supporting finetuning via LoRAs.

To qualify "insane speed" for anyone unaware, think a 10x improvement over even Cerebras. On the order of ~15,000 tokens/s. Not saying their approach scales well enough to keep pace with the various frontiers, but using their demo alone feels like a paradigm shift.

Yeah, I'm guessing this isn't unique, but I remember the first time I used ChatJimmy, I missed the fact that it had responded because I was still hitting the enter key, and getting ready to see tokens stream in, but they were already all sitting there, and I'd missed registering the visual diff somehow.

This seems like it would be an altogether different experience than the common experience of using an LLM, which is characterized by the person spending a lot of time waiting on the machine.

Yeah, it'd be a lot easier to maintain flow, less need to work on more than one session at once, etc. And then tool calls would be the limiting factor, especially network access. I hope AMD keeps the project moving forward post acquisition.

Yes and no. A single chip cannot have it's weights adjusted, once it's out, it is what it is.

But also, the model weights are in a single mask rom layer, high up in the metal stack. They could manufacture the die specialized for a given geometry of a model up to that layer, wait for updated weights, and then get the final product out in weeks after they got the weights, instead of many months which is what it would take to redesign the whole chip for the new weights.

Reply on news.ycombinator.com