Hacker News Reader

Comment by 0xfaded | Hacker News Reader

I have some CUDA code that I need to run as fast as possible, so if I was going to blow $1500, my use case would imply that I should go with two 3080s.

However, I also play around with deep learning stuff, expect to do so more in the future, but don't currently follow it so closely.

Would someone care to ponder on what difference they think a 24GB gpu vs a 10GB gpu will have as a tool for deep learning dev over the next 3 years?

For what it's worth, I'm a computer vision guy, but I did have a play with DeepSpeech earlier this year.

View on HN

Just use AWS or GCP. Here's the calculation:

Two 3080s = $1500 + number of hours times electricity cost (for me, this would be about $0.14/hour)

A K80 from AWS (close enough to the same RAM) = $0.45/hour (spot pricing) or $0.9 (on demand)

I've been able to use spot instances for basically all of my AWS hobby work, so I'll use that.

The crossover point where it's worth buying is where 1500 + 0.14h = 0.45h, about 5000 hours or 30 weeks of training.

For hobby work, do you expect to have it training for 20% of the next 3 years? That's all without considering the fact that you might want to upgrade to a newer card, AWS's prices will likely fall, and you might want a card with more RAM sometimes, but not others, or even a multi-GPU setup sometimes.

I used to spend money on fancy cards and machines and justify it with my hobby learning. Now I just let somebody else do the heavy lifting and pay them rent. It's spoiled me. You go from spending a few nights trying to get a thing performant to throwing a burrito worth of money at amazon to just parallelize the dumb thing on a massive machine and having it done by bed time.

I also used to justify buying the fancy card with my gaming hobby, but then when I actually did have a long running training job, it'd be super frustrating because now I can't use the desktop for anything else for a few days until training finishes.

I have to admit you're right. My case is a little different, where my jobs are about 1h long and it would take me longer to copy all the data to AWS than running it locally.

If you transition your workflow to AWS, and do preprocessing and storage and training in AWS, it avoids the issue. As a benefit, storage in s3 is dirt cheap compared something like dropbox, and archival is easy and cheap enough to just forget about. I feel weird like I'm evangelizing amazon, but it's a workflow i'm very happy with and trying to streamline to take even more advantage of.

> The crossover point where it's worth buying is where 1500 + 0.14h = 0.45h, about 5000 hours or 30 weeks of training.

1500 + 0.14h should be equal to 0.45h * 4. A 3080 will at the least be 2x faster for general training than one k80.

Might be worth pointing out that fp16 on a k80 is woeful.

Being in the ML space myself I can tell you that memory is pretty important. With my current workload (vision tasks) it is my largest constraint.

That said, rumor has it that they will announce a 20Gb 3080 later.

and its a 20gb with the ability to do FP16 right? So theoritcally lots of the models can actually be 40gb?

Well, they can be 40 GB fp32 models technically, but translating a model trained in fp32 to fp16 is not trivial (trust me, we’re working on this right now for a model). But remember that training the model requires a lot more memory than just the model parameters, because you need to store the gradients as well.

What the other commenter said is absolutely true, but what's even more important is that 3080 doesn't seem to support SLI so you're "stuck" with one 3090 with that budget (which seems to be the only card supporting SLI this gen).

Not an expert in ML, but I don’t think CUDA uses SLI at all.

SLI is specific to rendering. Depending on the workload, it’s sometimes makes perfect sense to split GPGPU jobs into multiple GPUs. An extreme example of that approach is crypto-currency miners who sometimes use a dozen of GPUs in a single computer.

The only limitation, the working set used by each GPU needs to fit in VRAM of that GPU, otherwise GPUs gonna bottleneck on I/O as opposed to compute, will be very slow. For ML, this means the setup of two 3080 GPUs will be limited to 10GB model sizes.

It's not impossible to distribute training across multiple GPUs, but it's certainly not straightforward. And if you want activations for an entire 4K image in the same model, having a lot of memory on one card is your friend.

I'm fairly lay but imo GPT-3 demonstrates pretty soundly that huge models are no magic bullet- it's got >2x as many parameters as the human brain has neurons and it can't do long division. Dogs and other animals get by just fine having less than 1% as many neurons as humans.

Even a billion parameters is a huge model, and a factor of 2.4x increase is not going to make a tremendous difference in your performance. In particular the data-heavy nature of vision stuff means that you'll be bottlenecked by training more than memory, AFAIK (again, lay).

This is a misleading comparison. You are comparing a massive model with huge models. What you should be comparing are big models vs medium models that a single consumer GPU will fit. And - you don't need to take my word for it, there's tons of papers - the bigger models definitely perform better.

Model parameters and biological neurons do not map 1:1.

The human brain has about 100 trillion synapses which is a closer analog to ML model parameters.

It can't do long division because it literally can't read the number inputs, due to the way text is encoded (BPE). That it manages to learn any sort of arithmetic despite that is pretty impressive to me.

Reply on news.ycombinator.com