Hacker News Reader

Comment by joshstrange | Hacker News Reader

I’d really love to build a machine for local LLMs. I’ve tested models on my MBP M3 Max with 128GB of ram and it’s really cool but I’d like a dedicated local server. I’d also like an excuse to play with proxmox as I’ve just run raw Linux servers or UnRaid w/ containers in the past.

I have OpenWebUI and LibreChat running on my local “app server” and I’m quite enjoying that but every time I price out a beefier box I feel like the ROI just isn’t there, especially for an industry that is moving so fast.

Privacy is not something to ignore at all but the cost of inference online is very hard to beat, especially when I’m still learning how best to use LLMs.

View on HN

> but every time I price out a beefier box I feel like the ROI just isn’t there, especially for an industry that is moving so fast.

Same, esp. if you factor in the cost of renting. Even if you run 24/7 it's hard to see it paying off in half the time it will take to be obsolete

You pay a premium to get the theoretical local privacy and reliability of hosting your own models.

But to get commercially competitive models you need 5 figures of hardware, and then need to actually run it securely and reliably. Pay as you go with multiple vendors as fallback is a better option right now if you don't need harder privacy.

With something like OpenRouter, you don't even have to manually integrate with multiple vendors

Is that like LiteLLM? I have that running but never tried OpenRouter. I wonder now if it's better :)

You can also try LangDB or Portkey

Yeah, really I'd love for my Home Assistant to be able to use a local LLM/TTS/STT which I did get working but was way too slow. Also it would fun to just throw some problems/ideas at the wall without incurring (more) cost, that's a big part of it. But each time I run the numbers I would be better off using Anthropic/OpenAI/DeepSeek/other.

I think sooner or later I'll break down and buy a server for local inference even if the ROI is upside down because it would be a fun project. I also find that these thing fall in the "You don't know what you will do with it until you have it and it starts unlocking things in your mind"-category. I'm sure there are things I would have it grind on overnight just to test/play with an idea which is something I'd be less likely to do on a paid API.

You shouldn't be having slow response issues with LLM/TTS/STT for HA on a mbp m3 max 128gb. I'd either limit the entities exposed or choose a smaller model.

Oh, I can get smaller models to run reasonably fast but I'm very interested in tool calling and I'm having a hard time finding a model that runs fast and is good at calling tools locally (I'm sure that's due to my own ignorance).

I decided on openai api for now after setting up so many differnt methods. the local stuff isn't up to snuff yet for what I am trying to accomplish but decent for basic control.

I use a combo of Anthropic and OpenAI for now through my bots and my chat UIs and that lets me iterate faster. My hope is once I've done all my testing I could consider moving to local models if it made sense.

> You don't know what you will do with it until you have it and it starts unlocking things in your mind

Exactly. Once the price and performance get to the level where buying stuff for local training and inferencing… that is when we will start to see the LLM break out of its current “corporate lawyer safe” stage and really begin to shake things up.

The juice aint worth the squeeze to do this locally.

But you should still play with proxmox, just not for this purpose. My recommendation would be to get an i7 HP Elitedesk. I have multiple racks in my basement, hundreds of gigs of ram, multiple 2U 2x processor enterprise servers etc.... but at this point all of it is turned off and a single HP Elitedesk with a 2nd NIC added and 64GB of ram is doing everything I ever needed and more.

Yeah, right now I'm running a tower PC (Intel Core i9-11900K, 64GB Ram) with Unraid as my local "app server". I want to play with Proxmox (for professional and mostly fun reasons) though. Someday I'd like a rack in my basement as my homelab stuff has overgrown the space it's in and I'm going to need to add a new 12-bay Synology (on top of 2x12-bay) soon since I'm running out of space again. For now I've been sticking with consumer/prosumer equipment but my needs are slowly outstripping that I think.

For what it's worth, looking at the benchmarks, I think the machine they built is comparable to what your MBP can already do. They probably have a better inference speed, though.

A Strix Halo minipc might be a good mid tier option once they're out, though AMD still isn't clear on how much they'll overprice them.

Core Ultra Arc iGPU boxes are pretty neat too for being standalone and can be loaded up with DDR5 shared memory, efficient and usable in terms of speed, though that's definitely low end performance, plus SYCL and IPEX are a bit eh.

Reply on news.ycombinator.com