This isn't correct. It's 100% feasible to run very capable models on consumer devices.
I wrote up a feasibility investigation last year: https://fleetwood.dev/posts/a-case-for-client-side-machine-l...
I wrote up a feasibility investigation last year: https://fleetwood.dev/posts/a-case-for-client-side-machine-l...
...versus running the best models available, in a few seconds, without using up the memory the main app you're using needs for running.
These are all mainly going to be run remotely for general consumer usage for quite a while I think.
Also it had better spend almost all its time doing nothing or it would kill my battery. Same as with my CPU.
The main point still stands though -- it's pretty useless if it takes a couple minutes to do what a server can do in a couple seconds.
I miss Aero, that shit was so cool...
Well that's the problem though, those models don't come any close to being useful at all. At least not yet. And they also run much slower.
As compute increases in general, there will be larger and more capable state of the art models and it'll make more sense to just use those instead of trying to run some local one that won't give you any useful answers. Data centers will always have a few orders of magnitude more horsepower than your average laptop, even with some kind of inference accelerator card.
Also, not an LLM.
Sounds a lot like most of my early programming experiments…
Though I’ve heard on good authority that the early programmers looked past being able to calculate ballistic charts and have done some interesting things with these “computer” things.
Trying out some prompts, maybe last I used SD my mistake was going with a lower resolution to speed up generation. I literally cannot get this one to make anything that isn't a weird blob at 256px and lower, but at 512px it works fine? Weird that it's so resolution dependant. I guess some proper stuff can be made at 1024px and above.
Your contention is that models will run on devices; but latent diffusion models have lower memory footprints (see: latent).
The hardware you need to run a good LLM is what, 10x more than a latent diffusion one?
They are not comparable.