Hacker News Reader

Comment by wmf | Hacker News Reader

I assume this is a million-dollar rack of custom chips so it's probably not coming to mobile any time soon.

Well, currently its entirely possible to run these models on iPhones. It's just not practical because it eats all the resources and the battery when slowly generating the output.

Therefore if Groq has achieved significant efficiency improvements, that its, they are not getting that crazy speed by enormous power consumption then maybe they can eventually build low power mass produced cutting edge fabbed chips that run at acceptable speed?

The thing is, I don't see any efficiency improvements. I see models running fast on very expensive hardware using techniques that don't scale down.

Care to explain? Are they using 10x energy for 10x speed improvements?

They're using hundreds of chips. Based on the data sheet I would estimate this demo uses 173 KW. It may be 100x energy to get 10x speedup.

100s of chips for who knows how many clients. The mobile phone will have to do calculations just for 1 client.

Yes, we pipeline requests so multiple users are being handled by the same hardware at one time.

Thanks for the clarification. So, would you say that Groq has a potential to have let's say OpenAI speeds on handheld devices at reasonable energy consumption? Or is that not really what this tech's strength is maybe?

The industry as a whole is a very long way away from that. The power requirements are too high for mobile.

I think the limitation is chip size/cost. SRAM is a lot less dense than RAM. According to Google this is typically used for registers and caches, which are only megabytes large.

Reply on news.ycombinator.com