Power of local LLMs especially with new Apple chips

twitter.com

38 points·mobilio··4 comments

Hey! I am the @karmedge who wrote this post on Twitter

I wonder what do you guys think about tool which will be able to automate actions on computer? Like clicking smth, automatically opening the window for me after I type in some request?

Something like hyperwriter assistant but for Mac. Very similar to open interpret but with accessibility option

I wonder what else is there in the space?

Something what I'd really like, would be:

1. You have an application open, let's use for this example Apple Notes 2. You ask it to summarize the "Recipe xyz" 3. It will search for the recipe 4. Open the note 5. "Copy" the text 6. Summarize it for you

This is a very simple example and maybe not "cool" enough. But would be great to give it any kind of interface and then do some more "advanced" (for an LLM) tasks.

I think this could be possible with GPT4-Vision, but it would be very costly, for things which you could do yourself with a few clicks and key presses.

An important missing piece in the price analysis is that the M3 chip is not designed for continuous load. It's the difference between retail and datacenter/commercial hardware. You can get an RTX-4090 for 1500. But if you try to train on it for ~a few months it will almost certainly die. There's a reason datacenter hardware has a 5-10x price premium.

Local LLM use isn't really continuous though. You're comparing Apples to oranges.

For training especially and for the datacenter, NVIDIA cards still make absolute sense. But for consumer use cases, and even for developers, it's not so cut and dry. Very comparable performance for short-term bursts of compute.

Reply on news.ycombinator.com