Hacker News Reader

Comment by armchairhacker | Hacker News Reader

If I am looking to buy a GPU chip for ML research:

- What chip should I buy? - When should I buy? Should I wait for prices to drop? Will new and improved chips be released anytime soon? - What are the advantages / disadvantages of each chip (3060 vs 3080, Nvidia vs AMD)? Which chip is most cost-efficient? What are each chips' specialties (e.g. specific type of neural network, graphics vs compute)?

View on HN

This applies for all neural networks. Depending on how much money you're willing to spend, in descending order: DGX (computer with 8 A100s, $150,000), A100 (80GB, $15,000), A6000 ($5000), RTX 3090 ($1000).

I'd consider renting these, especially if you are just getting started:

https://docs.paperspace.com/core/compute/machine-types/

1 x A100, 80GB is $3.19 / hour. 8 x A100, 80GB is about $25 / hour.

They have much less expensive machines. I used to use them to run steam games, but now proton is just too darn good. Their low end machines are OK for CAD software that supports real-time raytracing.

Lots of good options for cheap GPU clouds, including Paperspace (mentioned above), Coreweave, and Crusoe Cloud (crusoecloud.com). Crusoe Cloud's angle is that our GPUs are powered off otherwise wasted energy and are carbon-reducing; running one for a year provides an emissions reduction equivalent to taking one car off the road.

Disclosure, I'm head of product @ Crusoe Cloud. Feel free to ping me at mike at crusoecloud dot com if you've got questions or feedback.

How can this be carbon-reducing? That sounds like a dubious marketing claim.

TL;DR: we run data centers on-site at oil wells and take natural gas that would otherwise be flared (it can't be economically transported as natural gas in a pipeline or turned into electricity and transported) and combust it completely. Methane is a significantly more potent greenhouse gas than CO2, so it ends up being a net reduction in emissions vs what's currently happening.

https://www.crusoeenergy.com/digital-flare-mitigation has some more information.

Is flare combustion more incomplete than your generation?

on the other hand, does that mean you're obligated to be in oil producing countries? how does the geopolitics of that work?

Cool idea. Tried to join your waitlist but the form throws errors. Could be my security measures, but I’m on mobile safari, toggled off all content blockers and still had an error.

Odd, I just tested on Safari mobile and it went through without issues. Mind sending me an email at mike at crusoecloud dot com and we can get you set up?

How do you carbon offset?

We have a bunch of diseased pine trees. Assuming we don't have enough time to cut them down and burn them in the next 12 months, and that I am unscrupulous, I could sell you carbon credits through one of the bigger exchanges. You would never know, and it would be completely above board.

John Oliver had a particularly depressing segment on this recently.

(Totally off topic, but I'd love to biochar them instead. Using them for lumber and shipping them offsite are non-starters. Any ideas, anyone?)

See above post, but the TL;DR: is "we capture methane that would otherwise be flared."

Agreed that a lot of carbon offsets look like, "we were going to cut this section of the rainforest down, but if you pay us, we won't do that for X period of time." This is _actually_ reducing existing emissions.

As someone who just fired off both paperspace boxes and coreweave to play around with stable diffusion, Paperspace limited me to some old janky hardware and never replied to my request to "unlock" the more modern cards.

Coreweave let me get a modern card box first go round. I'd go with coreweave if you are a weekend hacker who just wants access to a lil bit of beefy GPU.

YMMV.

Are there "mining pools" for running AI jobs and getting paid in crypto?

There's vast.ai, but you get paid in USD.

Won't 3090 prices go down a good bit after the 4xxx series cards start coming out in a couple of months?

My guess is "not as much as most folks seem to think"

Comparable 24gb vram 4xxx cards are also 1 slot bigger and many watts hungrier. If you want to be able to use a 800w PSU + 3 slots, and just need 24gb vram for say, running inference on a big diffusion model, then 3090 is still going to be your only option for a while.

They're pretty well priced right now, at $1k. If you need one, not much reason to wait, your time is probably worth more than saving a couple hundred bucks.

Is there no consumer option bigger than 24gb next gen?

It appears as though the 4090 will be 24GB, but that card may also be almost $2k.

Used 3090s on eBay are $800 all day long. That price may drop a bit in the next week or so, but not much, as that 24GB of VRAM is the main draw for that over a 3080Ti.

The 4090ti is rumoured to be 48GB [1], but who knows when that will release or how much it will cost. If you really need extra VRAM and don't mind longer inference, older used Tesla cards are an option. A used Tesla V100 32GB can be sometimes found on Ebay for 1500.

[1] https://www.techpowerup.com/gpu-specs/geforce-rtx-4090-ti.c3...

Thank you great tip on V100.

I just grabbed an A6000 for $3500 on eBay, so you can probably get a pretty decent deal on those now. They're pricey but IMO it's a great deal if you really need the VRAM (e.g. for training LLMs).

Lambda Labs crunched the numbers in Feb 2022 [0]. They concluded:

“”” So, which GPUs to choose if you need an upgrade in early 2022 for Deep Learning? We feel there are two yes/no questions that help you choose between A100, A6000, and 3090. These three together probably cover most of the use cases in training Deep Learning models:

Do you need multi-node distributed training? If the answer is yes, go for A100 80GB/40GB SXM4 because they are the only GPUs that support Infiniband. Without Infiniband, your distributed training simply would not scale. If the answer is no, see the next question.

How big is your model? That helps you to choose between A100 PCIe (80GB), A6000 (48GB), and 3090 (24GB). A couple of 3090s are adequate for mainstream academic research. Choose A6000 if you work with a large image/language model and need multi-GPU training to scale efficiently. An A6000 system should cover most of the use cases in the context of a single node. Only choose A100 PCIe 80GB when working on extremely large models “””

[0] https://lambdalabs.com/blog/best-gpu-2022-sofar/

You should get a Nvidia card with as much VRAM as possible. A 12 GB RTX 3060 is probably the most cost efficient at the moment.

I don't think AMD is really viable for ML. Nvidia has the mind share in that segment, so nearly all tools will work with Nvidia, while very few support AMD.

Not disagreeing, but my < $300 8GB AMD runs stable diffusion just fine.

Guessing "just fine" is relative, but care to put a prompt + the settings you use, and how many it/s you get? I'm getting max ~8it/s on a 2080ti and I think even that feels slow sometimes so looking to upgrade my GPU now, curious to see what "just fine" means for you here.

100 seconds per batch of five (1.7it/s); default settings (512x512, etc)

Performance seems to be prompt-independent; I'm using a docker container that seems to have disappeared, this modified version for amd cards with less than 10gb:

https://github.com/basujindal/stable-diffusion

and this workaround:

https://github.com/RadeonOpenCompute/ROCm/issues/1756#issuec...

Anyway, don't "upgrade" to my budget AMD card. :-)

> 100 seconds per batch of five (1.7it/s); default settings (512x512, etc)

What settings precisely? Scale, steps and sampler being the most important ones (together with size, but you already shared that :) )

Stable Diffusion is the kind of phenomenon where people are actually contributing other backends (like the one for the M1 GPU). That’s not super common though, a lot of the time if you want to get a network running you need an Nvidia card so you can use CUDA (it’s not even about hardware performance, just that CUDA and CUDNN and so on are written by Nvidia for their GPUs).

I have a dumb question: when someone implements a backend like mps for stable diffusion, what are they actually implementing? Shims for Nvidia proprietary stuff that doesn’t exist outside of CUDA?

Keep in mind that bigger models are coming. And to use all features of even the current SD version, you need a lot of VRAM - 12GB for textual inversion (making it learn your own style), 30GB+ for Dreambooth (sort of micro-finetuning that doesn't need a GPU farm and a huge tagged dataset), a lot for img2img on a high-res picture. It also massively benefits from the large amount of compute cores.

Right now, the bigger and faster, the better. And there's really no limit of the computing power you can throw at various tasks to make them run better. It almost looks like 90s again.

I picked up a 3090 Ti for this purpose given the price drop. The 24 GB of VRAM is hard to beat.

Not much difference between 3090 or 3090 ti though, while the price is usually 30% higher.

Mostly because the 3090 Ti was discounted to $1,000, at which point it's about the same as the non-Ti version.

The prices are identical now from what I’ve seen.

The 3090 Ti also pumps out a lot more heat.

Needs a bit of an update but pretty good article

https://timdettmers.com/2020/09/07/which-gpu-for-deep-learni...

Reply on news.ycombinator.com