Hacker News Reader

Comment by jamesy0ung | Hacker News Reader

What exactly does the Xeon do in this situation, is there a reason you couldn't use any other x86 processor?

I think it’s that most non Xeon motherboards don’t have the memory channels to have this much memory with any sort of commercially viable dimms.

Pcie lanes

I was about to correct you because this doesn't use PCIe for anything, and then I realized Arc was a GPU (and they support up to 8 per machine).

Any idea how many Arc's it takes to match an H100?

I am reading from time to time about multi GPU solution and last time I found some real life information about this (it was two 7900 xtx) the result was that performance is the same at best often it is slower. So even if you manage to slap like 8 cheap cards onto motherboard, even if you would somehow make it work (people have problems with such setups), even if this would work continuously without much problems (crashes, power consumption) performance would be just OK. I am not sure if spending 10k on such setup would be better than buying 10k card with 40gbs of RAM.

Ollama works fine with multi-gpu setups. Since rocm 6.3 everything is stable and you can mix different GPU generations. The performance is good enough for the models to be useful.

The only thing which doesn't work well is running on iGPUs. It might work but it's very unstable.

Good to know. Still, is it viable option? Buying i.e. AMD Threadripper for 2.5k$, motherboard and ram for 2k$ and i.e. 4 GPUs for 4k$ to have total of 96GB of VRAM? Total should be around 10k$ which is roughly price of Intel GPU specifically for AI if I am not mistaken? Which option would be better performance wise. I did never saw a comparison anywhere and this is too much money to make fun experiment over the weekend.

> 10k$ which is roughly price of Intel GPU specifically for AI

Huh? The largest vram card that Intel has is the A770 which is around $350. What exactly are you trying to compare against? Are you doing inference only or training?

It can be read as a typo for "Nvidia" as their cards are about that price (or more, i haven't looked)

  DDR4 UDIMM is up to 32GB/module  
  DDR5 UDIMM is up to 64GB/module[0]  
  non-Xeon M/B has up to 4 UDIMM slots 
  -> non-Xeon is up to 128GB/256GB per node

Server motherboards have as many as 16 DIMM slots per socket with RDIMM/LRDIMM support, which allows more modules as well as higher capacity modules to be installed.

0: there has been a 128GB UDIMM launch at peak COVID

There's not much else (other than Epyc) in the way of affordably priced motherboards that have enough cumulative RAM. You can buy a used Dell dual socket older xeon CPU server with 512GB of RAM for test/development purposes for not very much money.

Under $1500 (before adding video cards or your own SSD), easily, with what I just found in a few minutes of searching. I'm also seeing things with 1024GB of RAM for under $2000.

You also want to have the capability for more than one full speed at minimum PCI-Express x16 3.0 card, which means you need enough PCI-E lanes, which you aren't going to find on a single socket Intel workstation motherboard.

Here's a couple of somewhat randomly chosen examples with 512GB of RAM and affordably priced. they'll be power hungry, and noisy. Same general idea from other x86-64 hardware such as from hp, supermicro, etc. These are fairly common in quantity so I'm using them as a baseline for specification vs price. Configurations will be something with 16 x 32GB DDR4 DIMMs.

https://www.ebay.com/itm/186991103256?_skw=dell+poweredge+t6...

https://www.ebay.com/itm/235978320621?_skw=dell+poweredge+r7...

https://www.ebay.com/itm/115819389940?_skw=dell+poweredge+r7...

PowerEdge R series is significantly cheaper if you already have an ear protection

yes, an R730 or R740 for instance. There's lots of used R630 and R640 with 512GB of RAM as well, but a 1U server is not the best thing to try putting gaming GPU type pci-express video cards into.

i imagine you can't use the pcie 1x risers on inference workloads, like you can for crypto mining, as there's no data going between the card and the CPU in crypto and i guess inference is heavily bandwidth restricted. Unfortunate!

The PCI-E slots in something like an 1U R630 or R640 are x8 or x16, the problem is more with the cooling and size/shape of the cards and how they cool themselves. The slots are meant to be used with things like 10 or 100 Gbps network cards or SAS/SATA host adapters which are considerably lower wattage than even a video card that's much weaker than an intel 770.

Commonly you will also find configuration with two or three 'low profile' pci-express slots which have a different card height than the 'standard' height that most GPUs are built at.

no, i mean like this https://www.youtube.com/shorts/rTCInAXSzKA

that works for crypto because all the CPU sends to the card is the target ledger sha256sum (simplified) and the GPU generates nonces until the `sha256sum(sha256sum(nonce += ledger sum)` has however many zeros in the front. So until a card finds the correct nonce, or the server sends "new work" - a new ledger shasum, there's no traffic, really, between the GPU and the CPU. housekeeping, whatever, but not like 1GB/s!

There are Xeon-based crypto motherboards on AliExpress with ~five x8 slots, I do sometimes wonder if those would work. NVIDIA Tesla K80/P40 24GB are on eBay for minimum $15 apiece. 120GB VRAM under $500 or so. Or maybe you could theoretically do a 96GB per node cluster with one x8 link for bottleneck free interconnect.

But it's likely never going to work, too many driver, compatibility, requisite Kernel development, and power issues, to name a few. Probably cheaper in the end to just go buy 5090 and rant about CUDA.

I had a couple of those boards. The full size slots aren't actually x8, because the CPU those boards support only have 24 PCIe lanes, and over half of them are just running USB, SATA, etc. The full size slots are so you can secure cards to the board, instead of running them zip tied to metal dish racks (like in the video i linked in my reply to your sibling comment: https://www.youtube.com/shorts/rTCInAXSzKA)

No, there are versions of those with recycled Xeon E5 and bunch of x8(or so advertised) slots unlike most LGA115x mining boards.

sorry, i completely missed "xeon" in your comment.

Reply on news.ycombinator.com