Hacker News Reader

Comment by newsclues | Hacker News Reader

Are we at or approaching the point where people can self host an useful LLM for personal use on affordable hardware?

I run Yi 34B at 45K-70K context as a GPT 3.5 replacement on a Nvidia 3090.

Its quite smart, and fast.

The whole rig cost me $2.1K, but it could have easily been $1.5K without splurging on certain parts like I did. And its 10 Liters, small enough to move around.

Same, but mine is 11L (A4-H2O). Which case did you get?!

A Node 202! I wanted something flat enough for a suitcase.

https://www.reddit.com/r/sffpc/comments/18a7mal/ducted_3090_...

Is a MacBook Air affordable? Because we’re already at that point.

The base model?

Given the article about GPUs, I think I was thinking more along the lines of a desktop with a consumer GPU.

Intel (and AMD) reportedly have big M-Pro-like iGPUs in the pipe, and Intel has the Battlemage GPU line coming up, so we shall see.

By base model do you mean off-the-shelf? I think the bigger limitation there would be RAM, as the machine comes with just 8GB soldered on board. You can expand to 24GB for +$400 though, and increase the GPU cores +25% for an extra +$100.

The CPU/GPU speed of the Air is the same as the MacBook Pro base model though. The only difference is the lack of active cooling, which for large workloads can result in performance degradation. Chat apps are intrinsically interactive though, only using bursts of GPU when it is performing inference. Shouldn't be an issue.

The expensive PRO/MAX variants of the MacBook Pro would be 2x - 4x faster. But the plain M2 in the MacBook Air is sufficient to get real-time speeds on sizable models.

When the M3 comes out for the MacBook Air, discounted M2 machines, new or refurbished, should be an ideal entry-level machine for local inference.

> discounted M2 machines, new or refurbished, should be an ideal entry-level machine for local inference

At the point of purchase of the lowest cost configuration with 24GB Unified Memory, you've already paid the an equivalent of over 2200 hours of GPU compute time on an RTX 4090 24GB, with a performance that exceeds the MacBook by around 1200% (it/s).

If you buy that MacBook for AI, you would have to run continuous generative inference on it for over a decade to match the return of just having used runpod instead. Even Apple doesn't use Macs to do AI - they use GCP TPUs. Buy the Mac if you like it, by all means, but be realistic.

LLM performance on Apple Silicon is decent for small models but does not scale. If any Mac architecture were cost effective at any scale for AI, we would be putting them in data centers, and we're not.

This sub thread isn’t about hosted inference though. I interpreted the question as “is there a reasonable laptop out there that can run large models locally in real time?” And a MacBook Air does fit the bill for that.

It’s an entry-level Mac, and it is capable of doing inference jobs. If you have around $1k available to spend on a daily driver laptop, but you also want to do some AI inference experiments, the MacBook Air is what I’d recommend.

The AMD Framework laptop might be a good alternative. But I don’t know how well its integrated GPU is supported by llama.cpp.

If you use Windows and you have a discrete GPU from any vendor with at least 6GB VRAM, you could test my implementation of Mistral model: https://github.com/Const-me/Cgml

Screenshot: https://github.com/Const-me/Cgml/blob/master/Mistral/Mistral...

Consumer laptops, such as apple silicon macbooks with just 32gb ram, can run fairly large models, such as Mixtral-8x7b, an open source model that's comparable to gpt-3.5, well enough for interactive chat, (and seems likely models of this size will keep improving) - ofc, 64gb ram preferable, because it unlocks larger models. And 3090s are the other way to to go, which are faster but not quite as much vram, though you can use multiple.

I'm using GPT4ALL with Mistral on a $400 Intel NUC, so yes, if the use fits!

Reply on news.ycombinator.com