Hacker News Reader

Comment by Abishek_Muthian | Hacker News Reader

Which CPU gen are you suggesting, is there any writeup on such setup where <10K (not incl. power bill) cpu only rig is giving usable token speeds on latest SoTA open weights models?

In my experience with rig half that cost, entire exercise of running coding models locally has been a huge disappointment.

Cost/Value when compared to cloud services is just not there, but I see the merit for those who value privacy over quality of output and want a backup of huge condensed corpus of data within their control.

Kudos to OP though, They had clear goals and they achieved it.

View on HN

I realized I didn't answer the CPU question, as a very quickly chosen example from eBay, there's a Dell R740XD with two Xeon Gold 6254 CPUs, 768GB RAM for sale for something like $5799 USD right now. I'm sure if I put some more time into it I could piece together something with a full terabyte for around the same price. Or faster/better CPUs, more core count CPUs by buying the system with no RAM, or minimal RAM (64GB) and then adding the DIMM kits from the more reputable refurb server part vendors on ebay.

It won't be fast at all, for certain, but it'll have enough memory to prove a configuration and be able to really use gargantuan GGUF format LLMs in the latest compiled llama-server. Re: electricity, I pay the equivalent of $0.07 ro $0.09 USD per kWh so it's not an extreme burden to have a theoretical 500W server running. Something like $35 to $50 of electricity a month if it's 500W 24x7.

Xeon Scalable in general seems like a good idea due to 6-channel (relatively) inexpensive RDIMM memory, but I've been reading that NUMA kills inference performance. Anyone got experience with multi-socket systems? IIRC even within the socket these cpus are divided into sub-numa nodes.

Even though LLM benchmarks are very opinionated, I would really like to see some numbers for the setup parent suggested. From what I read elsewhere, anything below $40K in HW costs is not worth the effort for coding models locally.

The old Cascade Lake based server found by the previous poster is still new enough to have instructions for relatively fast AI inference with the INT8 format.

So for optimal speed the models must be quantized in this format.

It is very likely that with INT8 models those CPUs are fast enough so that the inference throughput is limited by the memory bandwidth (384-bit interface to DDR4-2933 per socket, i.e. 282 GB/s for both sockets).

The memory throughput for such an old server is very similar to an AMD Ryzen Strix Halo, NVIDIA DGX Spark or Apple M5 Pro, but it has much more memory.

The inference speed should be very similar to those, but with bigger LLMs.

Would be nice if you could somehow connect GPU-levels of parallel floating point cores to that amount of memory. I guess that's what the big AI datacenters are doing, but how can we do that on a budget?

I think there is a good sized population of people who absolutely don't want to submit everything they do to an off site service, or let their content be used for unknown training purposes, and will tolerate slowness at 1 to 10 tok/s as a tradeoff.

Or people who want or need to run an uncensored (abliterated) gguf file to deal with controversial topics that a paid LLM service will refuse to work with or ban you for.

Not just controversial but also regulated areas. Virtually every law firm would be interested on locally-hosted AI at a reasonable price. So too ever medical research lab. Every CGI firm doing work for film/TV. And all the video game developers.

Do they care about locally-hosted, or only about self-hosted? I'm not really clear why a local box would be any better than running on a private AWS instance in any of these scenarios...

For one, doing the math on what it costs to rent a 768GB+ RAM AWS system with 40+ high performance CPU cores makes it very unappealing to pay for 12, 24, 36 months of it.

The largest high performance compute ec2 offering, the c9g.metal-48xl , maxes out at 384GB RAM and already costs a shitload.

The m9gd.48xlarge and m9gd.metal-48xl both have 768GB RAM and I cringe to think what they cost monthly. I just did the math on one of these and it costs $12 per hour, or $289 a day, or $8900+ for one month.

Also plenty of Europeans or people from other locations may consider it as an unacceptable risk factor to put their "off site" self hosted AI stuff with an American controlled company. Particularly if the servers are physically in the USA.

Hetzner will also rent you 768 GB of RAM with a Blackwell 6000 Max Q GPU for €2300/month [1].

Yes, it's a boatload of cash, but that's a €13,000 GPU and €20,000 of RAM at present prices. There is a segment of businesses where a fixed €28k/year bill is going to be preferred over plonking down €40k for a (theoretically) depreciating asset and ongoing colocation costs.

[1]: https://www.hetzner.com/dedicated-rootserver/gex131/

Renting something at a rate that'd be purchased in less than 2 years seems very myopic to me. And yeah it depreciates, but not to zero. So if you're speaking of the breakeven point after liquidation, you're probably there in well under a year at those rental prices.

> Renting something at a rate that'd be purchased in less than 2 years seems very myopic to me

And yet basically all AWS customers are doing exactly that. Turns out that making CAPEX "someone else's problem" is worth quite a lot to many businesses

that would be implying that "private" really means anything for AWS. Because if it's "private" as in "private" github repos that were totally not used for training copilot because they said so or "private" claude chats that are totally scanned even if you have enterprise contracts to check you are not doing anything malicious or are from china or whatever, and this will totally not be used for training...

can we trust any US based service to guarantee privacy and confidentiality? especially to us european frienemies?

> that would be implying that "private" really means anything for AWS

Insert your dedicated hosting provider of choice for 'AWS' (somewhere like Hetzner will be cheaper anyway).

But in general, AWS hosts are yours, running your code, with your security policies enforced. Sure, the US government can silently subpoena the contents thereof, but aside from that fairly extreme case, it's not like AWS is handing your data over to 3rd parties.

I would suspect that one would buy based on mem-bus & PCIe bus speeds more than CPU for this, and just dial down the CPU parameters to save power. Most of the time and power will be consumed by memory and bus transfers because the CPU will mostly be waiting to the right set of weights and factors to multiply.

Reply on news.ycombinator.com