Ask HN: Best single GPU and all-purpose server? | Hacker News Reader

Ask HN: Best single GPU and all-purpose server?

15 pointsuniqueuid18 comments

Hi, hoping to benefit from your collective wisdom and experience here.

I need to buy a rack-mountable machine (ideally one) that has at least one decent GPU (A6000 or A100), a good CPU (thinking Epyc or Threadripper) and > 40TB of storage.

Usage will be all over the place - medium-sized offline (batched) data processing, deep learning with language and visual models, inference, and live data ingestion. Possibly all at the same time.

What should I buy? Budget is 20-40k.

I'm wondering in particular: - Is the HBM2 memory on an A100 worth the 2x price over a A6000?

- Should I go for a tower (such as the Lenovo P620) or a 2U rack? (space and power are irrelevant to me)

- Are there hidden costs with major suppliers (I really don't want to pay $1k for a 4TB dell hdd)

- Are there any gotchas with report to lights out management? I.e. can I remote reboot a threadripper machine?

Thanks!

View on HN

I bought such machines in an academic high performance computing center a few years ago, with similar budget. Here are a few thoughts:

- You definetly want to do rack mount. It is more extensible and future proof.

- You will get a server grade mainboard which features industry standards such as IPMI remote managament, redundant power supplies or SAS (which allows for extending hard drives, for instance).

- You should probably put as many RAM into this machine in this machine as you can afford, since this will be a limiting factor, especially if you do "all at the same time"

- With your budget, you can consider buying a few less powerful compute nodes (think of 5k-10k per node) which can be more suitable if you divide the computer to multiple users.

[edit:formatting]

Thanks, this is actually for an academic setting, so a good match.

I like the "as much ram as you can get" recommendation, and will probably aim for ~512GB which is a cost-effective maximum.

The thought of splitting it into multiple machines has occurred to me, but I'm discouraged by anything that creates distributed problems.

There is MPI, but it's cumbersome and performance depends highly on the network (I guess I'll have 10G).

Batch management such as slurm works, but also has quite some overhead. Some things are near impossible to distribute, i.e. some algorithms with large matrices. That's why a single e.g. 64 core machine seems more attractive.

But I'll take to heart all the recommendations for rack-mount - seems this is a way to prevent a lot of headaches.

If you can buy consumer grade GPU i'll for sure prefer that as you can buy 4x rtx 3090 in same price as one A6000 and for almost all use cases outside extremely big models it should be a lot better solution. (I have one PC with 4x rtx 3090 for DL and it is enough but sometimes as i need to run very big model i just turn on colab with TPU [inference] or rent A100 server as it'll be still cheaper then any other solution)

For CPU I cant say much cause we are always limited by PCIE speed in our use case and Xeons are idle most of the time :P

Thanks, that's a good data point.

I'm wondering especially what the tangent for future developments will be. Most of the large language models are out of my league anyways (e.g. the new yandex russian-english model was trained on 800 A100s and needs 200GB of GPU ram to fine tune).

So maybe it would be more effective to go for high speed instead of large capacity. But then you probably end up with a custom chassis and PSU since 4x 400 watts are not something that you can use on most off-the-shelf workstations.

Where do you plan to keep the machine? Is noise a concern?

If you plan to spend a lot time around it, then I suggest getting some kind of tower. Otherwise rackmount all the way. There are so many benefits to rack machines, here are a few:

* Superior cooling

* Enterprise reliability

* Built-in remote access system (so you can remotely cycle the power in the rare event of a hang)

* Easy to open up and work on

* Tons of HDD slots

Downsides: Rackmount machines tend to be varying degrees of insanely loud.

Is 2U a hard limit on the size?

I love me a Supermicro rackmount. They are the most cost-effective good quality servers that I know of. Overall quite reliable, easy to work with, easy to source replacement parts for in the rare case a component croaks.

GPU lines:

https://www.supermicro.com/en/products/gpu?pro=pl_grp_type%3...

Please follow up and let us know what you end up getting!

Off topic: Are FatTwins still the big kahunas of high-performance, cost-effective enterprise rackmount servers?

Final food for thought: HN has a lower concentration of hardware nerds than the homelab subreddit. You might like to ask there as well, you'll get better advice.

https://old.reddit.com/r/homelab/

I would second the recommendation for Supermicro. I've had good success with many of them over the years. Good price, above average reliability, very wide range of options, decent post-sale support.

Regarding the follow-up, I can at least tell you my current candidate pick:

The Lenovo P620 has new Threadripper Pros with their high single-core performance, can take ordinary (i.e. non-blower-style active) GPUs, take 2 PCIe4 NVME ssds plus ~8 PICe3 SSDs via lane splitting and have space for ~4-5 HDDs.

Putting one of these in a rackmount kit would be a pretty good choice balancing performance. Plus I can liberally use regular hardware that's not gold-plated. This should open some cheap upgrade options in the future, i.e. a new Hopper GPU.

The only downsides are (1) RAM limitations (although the max capacity is 1TB with the new 128GB RDIMMS) and (2) power, since it only has a 1000W PSU.

I posted a top level comment on my experience with the P620; please take a look.

Thanks, I had never seen FatTwins so far.

Looks like the chassis wants to run on 8 nodes fully equipped - I think that's outside my budget and would have me manage 8 separate systems.

I'm looking for a single beefy system to skip the management and ops overhead. But it looks like there are few chassis with space and thermal capacity for a 300W GPU AND many 3.5" slots at the same time.

Yeah, I only mention the FatTwins as they are ridiculous. Don't know if they can even house GPU compute.

The 4U supermicro GPU chassis looks pretty good, and accepts up to 8+TB ram.

Something like:

https://www.supermicro.com/en/Aplus/system/4U/4124/AS-4124GS...

https://www.supermicro.com/en/products/motherboard/H12DGO-6 (2x nvme m.2)

Though I usually like to have some regular PCIx slots so I can attach independent SATA HBAs.

Also consider:

https://www.reddit.com/r/HomeDataCenter/

I don’t have any direct advice, but I had a P620 for quite a while until Lenovo bricked it through a botched firmware they pushed.

If you have issues they prioritize new builds over service; my parts went from 2 weeks to 10+ and they finally gave me a huge credit on a Thinkpad because I needed something for work.

So for customer satisfaction I’m happy; the laptop is fitting with my new situation better. The machine was awesome. If something goes wrong plan on it being down for a LONG time, and I wasn’t the only one hit by one of these (there have been a couple) firmware bugs.

Regarding cooling: I designed and printed an upgraded extractor fan (92mm to 120mm) and saw a 15c drop in temps on a SINGLE loaded 3080. Do NOT plan on loading this up and running it with non blower cards without a second intake (there is a spot for a second 92mm) and an upgraded exhaust.

Thanks, incredibly valuable advice.

I do have some experience with a P520 (Xeon), but that had a blower-style card and not really problems with the thermal load.

Cool to hear that you can do something to help with airflow!

Do hardware research. I mean just look into workstation boards and the CPUs and the drives and all that and see how much it costs to put together yourself. If you don't need support and warranty then you might not even need it to come pre-built.

You just need one mega machine from the sounds of it and 40K seems like way more than enough to throw together a system that has a remote management capable mobo and all that compute power and drive storage.

Maybe also ask around at the Level1Techs forum https://forum.level1techs.com/

> I really don't want to pay $1k for a 4TB dell hdd)

Good thing HDDs do not cost that much at all

Also, it sounds like you want this... For yourself? At your own place? At your own building? Where will this be used and who is paying? That factors into this.

Thanks for the tips. I'll buy this for an academic setting, so self-assembly is out of the question unfortunately (procurement rules prevent that for the most part).

I was somewhat shocked to see that HDDs and RAM actually carry an insane premium when you get them from large corps - RAM easily costs > $500 per 64gb, which is almost double the market price. But I've also had bad experiences with HP, which seems to limit the selection of RAM sticks you can use to prevent third-party purchases.

What about using a reseller that sells 'refurbished' enterprise rack mounted servers? Years ago I bought quite a few machines from xbyte.com

I'd probably consider that for a personal machine, but procurement rules will probably prevent me from doing that because I need warranty and service contracts.

GTX 1080 TI

Reply on news.ycombinator.com