Hacker News Reader

Comment by the__alchemist | Hacker News Reader

I believe this is a CPU/GPU vs ASIC comparison, rather than CPU vs GPU. They have always(ish) coexisted, being optimized for different things: ASICs have cost/speed/power advantages, but the design is more difficult than writing a computer program, and you can't reprogram them.

Generally, you use an ASIC to perform a specific task. In this case, I think the takeaway is the LLM functionality here is performance-sensitive, and has enough utility as-is to choose ASIC.

View on HN

It reminds me of the switch from GPUs to ASICs in bitcoin mining. I've been expecting this to happen.

But the BTC mining algorithm has not and will not change. That’s the only reason ASICs atleast make a bit of sense for crypto.

AI being static weights is already challenged with the frequent model updates we already see - but may even be a relic once we find a new architecture.

We can expect the model landscape to consolidate some day. Progress will become slower, innovations will become smaller. Not tomorrow, not next year, but the time will come.

And then it'll increasingly make sense to build such a chip into laptops, smartphones, wearables. Not for high-end tasks, but to drive the everyday bread-and-butter tasks.

The world continues to evolve, in a way that requires flexibility - not more constraints. I just fail to see a future where we want less general purpose computers, and more hard-wired ones? Would be interesting to be proven wrong though!

TPU usb-c dongle is less than $100 (widely used for detecting people in home assistant / frigate nvr camera feeds). If one-off $100 purchase can replace (and improve 10x by speed) anthropic subscription even for 12 months - I don't see why not.

Sounds to me like there’s potential to use these for established models to provide cost/scale advantage while frontier models will run in the existing setup.

IME llama et all require LoRA or fine-tuning to be usable. That's their real value vs closed source massive models, and their small size makes this possible, appealing, and doable on a recurring basis as things evolve. Again, rendering ASICs useless.

Read the blog post. It mentions that their chip has a small SRAM which can store LoRA.

Neither the blog nor Taalas' original post specify what speed to expect when using the SRAM in conjunction with the baked-in weights? To be taken seriously, that is really necessary to explain in detail, than a passing mention.

Heh, I said this exact thing in another thread the other day. Nice to see I wasn't the only one thinking it.

The middle ground here would be an FPGA, but I belive you would need a very expensive one to implement an LLM on it.

FPGAs would be less efficient than GPUs.

FPGAs don’t scale if they did all GPUs would’ve been replaced by FPGAs for graphics a long time ago.

You use an FPGA when spinning a custom ASIC doesn’t makes financial sense and generic processor such as a CPU or GPU is overkill.

Arguably the middle ground here are TPUs, just taking the most efficient parts of a “GPU” when it comes to these workloads but still relying on memory access in every step of the computation.

I thought it was because the number logic elements in a GPU is orders of magnitude higher than in a FPGA, rather than just processing speed. And GPU processing is inherently parallel so the GPU beats the FPGA just based on transistor count.

With FPGA you are sacrificing performance for flexibility you are far less efficient in transistors for any given task than with a dedicated ASIC even if it’s a general compute ASIC like a GPU is today.

The reason no one is building large FPGAs is that there is no market for them.

If an H200 scale FPGA was viable we would have one.

Reply on news.ycombinator.com