Hacker News Reader

Really cool to see this! Local is the real future of AI.

I got really excited about this and fired it up on my petite little M2 Macbook Air only for it to grind it to a halt. Think the old days when you had a virus on your PC and you'd move the mouse then wait 45 seconds to see the cursor move. It honestly made me feel nostalgic. I guess I have to taper performance expectations with this Air, though this is the first time it's happened.

View on HN

Just wait 10 years when computers have a dedicated AI-PU and you don't have to worry about freezing anything up to talk to your bot.

The M1 line of Macs does have that unit.

I wonder if edge tpus (sold here https://coral.ai/products/) could help, though I guess the community would have to optimize for them instead of for standard hardware

How much memory do you have in your Macbook? The 7B models seem to work well with at least 16GB of unified memory, but I’ve seen Macs with 8GB really struggle.

Indeed it's just a poor little 8GB of RAM.

Reply on news.ycombinator.com