Hacker News Reader

Comment by mrtksn | Hacker News Reader

Does this make it practical to run LLMs on mobile devices? I wonder about the power consumption and if it can make sense to have it integrated in some future mobile devices. Or maybe have a dedicated storage, RAM and processing cores that goes as an USB-C add-on? A case with integrated battery and this chip?

I'm dreaming of having LLMs on anything. Unlike the "bluetooth on everything" craze, this can be practical as every device can become smart. Remember how some British researchers made a self driving car using an LLM? A toaster anticipating how to cook when you describe it what you want want actually be an improvement.

View on HN

I assume this is a million-dollar rack of custom chips so it's probably not coming to mobile any time soon.

Well, currently its entirely possible to run these models on iPhones. It's just not practical because it eats all the resources and the battery when slowly generating the output.

Therefore if Groq has achieved significant efficiency improvements, that its, they are not getting that crazy speed by enormous power consumption then maybe they can eventually build low power mass produced cutting edge fabbed chips that run at acceptable speed?

The thing is, I don't see any efficiency improvements. I see models running fast on very expensive hardware using techniques that don't scale down.

Care to explain? Are they using 10x energy for 10x speed improvements?

They're using hundreds of chips. Based on the data sheet I would estimate this demo uses 173 KW. It may be 100x energy to get 10x speedup.

100s of chips for who knows how many clients. The mobile phone will have to do calculations just for 1 client.

Yes, we pipeline requests so multiple users are being handled by the same hardware at one time.

Thanks for the clarification. So, would you say that Groq has a potential to have let's say OpenAI speeds on handheld devices at reasonable energy consumption? Or is that not really what this tech's strength is maybe?

The industry as a whole is a very long way away from that. The power requirements are too high for mobile.

I think the limitation is chip size/cost. SRAM is a lot less dense than RAM. According to Google this is typically used for registers and caches, which are only megabytes large.

I don't think we've put a GroqChip in a mobile device yet. Interesting idea!

> Remember how some British researchers made a self driving car using an LLM?

No? Do you mean actual, full self driving on normal roads in traffic?

Yes, IIRC they reason on the car actions using LLMs. They still use image processing but once you identify the objects in the scene, the LLM interprets and decides what to do with the car.

I'm not sure which one was it though(Ghost Autonomy maybe?).

Do you have a source? Because that actually, properly working would be headline global news and would value the company in the billions.

It was discussed here on HN, that's how I know about it.

I found a few things when searched around but not sure which one was the one I recall.

Anyway, here is a video from one: https://www.youtube.com/watch?v=C2rbym6bXM0

Here is a paper discussing something similar: https://arxiv.org/abs/2307.07162

The description for that video says

> Ghost Autonomy’s MLLM-based capabilities are currently in development. These video and image examples show MLLM-based analysis of driving scenes captured from Ghost vehicles driving in both autonomous and conventional mode. MLLM-based reasoning is not yet being returned to the car to impact actual driving maneuvers.

So the model discussed is not doing any driving whatsoever. This is not self-driving at any level.

Then its not the one I remember maybe.

Yeah just offload the compute onto the cloud.

Its too unreliable, too restricted and too not-private.

ChatGPT stopped processing images for me, trying to get help but support doesn't appear to be very fast, they asked for more info but not heard back since.

Its too restricted, can't do anything on hard topics. It doesn't work when you try to work out exploits or dangers in a system for example.

Its not private, they say they don't train on API requests but companies steer clear when it comes to send sensitive data.

The model being too restrictive does seem to be a good point.

Do you think there are less restrictive models hosted on poe.com?

Reply on news.ycombinator.com