Which GPUs to Get for Deep Learning | Hacker News Reader

Which GPUs to Get for Deep Learning

timdettmers.wordpress.com

119 pointsetiam38 comments

Open articleView on HN

"TL;DR advice

I work with data sets > 250GB: GTX Titan eBay

I have no money: 3GB GTX 580

I do Kaggle: GTX 980 or GTX 580

I am a researcher: 1-4x GTX 980

I am a researcher with data sets > 250GB: 1-4x GTX Titan eBay

I never used deep learning before: 3GB GTX 580 eBay

I started deep learning and I am serious about it: Start with one GTX 580 and buy more GTX 580s as you feel the need for them; buy new Volta GPUs in 2016 Q2/Q3

I want to build a GPU cluster: This is really complicated, I will write some advice about this soon, but you can get some ideas here"

I don't understand the claim that you should get a large RAM GPU if you have a large dataset. It's the model and your training procedure that dictates the amount of GPU ram consumed (e.g. fft convolution kernels like fbcunn use a decent amount of RAM compared to CuDNN's fused gemm, larger batch sizes mean more ram for storing activations, etc).

With a larger dataset a larger model is more useful though, since you are less likely to suffer from overfitting.

I've been curious about the Xeon Phi. It seems like they would fit the same purpose as the GPU (especially with the Christmas sale last year that you can get a 31s1p for ~$120-170), it has more memory and similar FLOPS/price. Why are they not as competitive as GPU?

Bluntly, it's a great processor on paper, but for the most part, its performance falls severely short of its paper specs.

While I was really looking forward to Intel shipping a competitive processor to NVIDIA's GPUs, reality disappointed. Intel put 2 engineers on porting a piece of code I wrote for a year. That code is roughly 20x faster on a Maxwell class GPU than on a Haswell class CPU (all cores firing). 2 man years later, instead of the Xeon Phi performing on par with a C1060/GTX 285 as it initially did, it managed to accelerate the CPU code by ~10%. In that time, NVIDIA shipped GK104, GK110 and then GM204 accelerating my code by a factor of 2 relative to GK104. When I tried to help one of the Intel engineers catch up to at least Fermi class GPUs, he got mad at my algorithmic choices and stormed out of the room.

Intel IMO will resort to dumping these things at rock bottom prices into data centers in order to spoof just enough low information government wage HPC administrators into believing these things are good for anything more than playing Jenga with them. And that will keep NVIDIA from achieving total HPC dominance in the short term.

No idea what the long-term will bring. Intel ought to be able to deliver a compelling part one day, but I suspect that will be a CPU with wider SIMD and more cores.

First that would require adapting the deep learning libraries (e.g. theano, torch and Caffe) to use it efficiently. Those libraries have optimal CUDA kernels for efficient GEMM, spatial pooling and FFT convolution operations that would need to be rewritten to leverage the capacity of a Xeon Phi chip.

I wonder if there will be a time when we can stop calling these things "GPUs", because right now, with all these new applications, it really doesn't make much sense anymore. Also, I hope that there will be a possibility to chain more of these cards together, because currently they hardly fit inside a regular machine (these cards are quite bulky with all their fans and heatsinks and stuff), and the PCI connectors are commonly quite closely spaced (and there are few of them). Adding or removing a single card is no joy. A server-rack unit has even less space. I'd rather see these cards made into their own (stackable) boxes, to be placed next to or on top of the main computer.

using cables for high speed links (~6GB/s) is hard, and whilst there are a few problems that do not require much host/device communication, there are a significant number that do - any increase in latency/decrease in bandwidth will be very noticeable in codes that otherwise perform well on accelerators (GPUs, etc.).

Weird that there's not a single mention of AMD cards.

My experience has been that none of the major Deep Learning libraries (Theano, Torch7, Caffe) offer support for OpenCL, whereas they all make it trivially easy to get models running on a CUDA GPU. On top of that NVIDIA has a library of deep neural network primatives[0], and I don't believe AMD offers anything similiar.

The general consensus I've seen is to just get an NVIDIA card if you're serious about working with deep neural nets on the GPU.

One thing that did surprise me was that there was no mention of using EC2 GPU spot instances for getting your feet wet. If you don't have access to a GPU with CUDA support you can get a spot instance for about $0.07 an hour to at least test out that you have your GPU code configured correctly (and you will see some performance gains). There are even a couple of AMIs out there with Torch7 and Theano already installed.

0.https://developer.nvidia.com/cuDNN

Thanks for elaborating, these are exactly the points why AMD's GPUs are just not used in deep learning.

AWS is great if you want to use a single or two separate GPUs. However, you cannot use them for multi-GPU computation as the virtualization cripples the PCIe bandwidth; there are rather complicated hacks that improve the bandwidth, but it is still bad. Everything beyond two GPUs will not work on AWS because their interconnect is way to slow.

That was my first thought as well. I know Nvidia pretty much dominates the GPU deep learning scene, but I talked to some devs who said they got more bang for their buck using AMD GPUs. And it seems their docs have gotten significantly better over the years.

Maybe they just prefer using CUDA over OpenCL? Perhaps AMD can change that around when HSA is supported across all of its product lines.

Why I did not mention AMD cards: NVIDIA's standard libraries made it very easy to establish the first deep learning libraries in CUDA, while there are no such powerful standard libraries for AMD's OpenCL. Right now, there are just no deep learning libraries for AMD cards -- so you just cannot use AMD cards for deep learning.

i feel that you should have at least mentioned it, otherwise it feels like the article implicitly asserts that only NVIDIA GPUs exist, or perhaps even that you are some way not impartial (partial..? that sounds strange).

Thanks for your feedback. I updated the blog post with a small NVIDIA vs. AMD section.

That and Nvidia's OpenCL implementation is almost the bare minimum you can do to say you have one (unless things have changed significantly within the past year or so). Improvements to OpenCL libraries are unlikely to come from anyone already locked into using their hardware.

Sounds like an opportunity for someone.

say that in the article

The libraries all use CUDA right now. Hopefully people will eventually have more OpenCL stuff.

AMD cards are slow, for these sorts of applications.

AMD cards were much faster than the Kepler equivalents, but now NVIDIA fixed their mistakes in that architecture and they are as fast as AMD cards. So AMDs hardware is good, it is just a software issue.

Was Kepler actually slow, or did it just seem that way because the 680 was built with their mid-range chip while the big high-end GPGPU-oriented chip was initially reserved for the server products?

Tesla GPUs are also rather slow, but the value of Tesla lies in increased double performance, larger memory and memory correction -- and you cannot beat Tesla GPUs in these departments.

Care to offer any benchmarks or an explanation of the architectural deficiencies that would make them dramatically under-perform their FLOPS and memory bandwidth ratings? When AMD cards were vastly preferred over NVidia for bitcoin mining, it had a very simple and straightforward explanation.

> Another important factor to consider however, is that the Maxwell and Fermi architecture (Maxwell 900 series; Fermi 400 and 500 series) are quite a bit faster than the Kepler architecture (600 and 700 series);

Where are the citation for this? I have heard about the disappointing benchmarks when the 600 serie launched, but this is uncommon as far as I know in the GPU marked to have sub part drivers on launch. Plus if you are following a tic-toc strategy like Intel does, you should rarely go for the tic iteration in my opinion. So is this just a claim from the author or is there something to back this statement up?

There are several cryptocurrency mining benchmarks which are bandwidth bound; those benchmarks reflect almost one to one the deep learning performance of a GPU. You can see this for example for litecoin mining: https://litecoin.info/Mining_hardware_comparison

Bought a 970 when they first came out, because this guy said they were probably going to be the card to get in a comment.

Completely my fault, but still disappointing

If you are dissatisfied with the 970 you might be able to get a refund: NVIDIA advertised specs that differ from the real thing – that is a good reason to demand and get a refund.

It's probably still one of the most cost-effective cards, even if the last 0.5GB of memory is less useful.

Less useful is an understatement: your working set simply cannot touch that memory region at all without crippling performance.

That's true of any system. A "real" 4GB that touches the 4.1GB region will be forced to use CPU main memory over the PCIe bus.

So the 0.5GB region is still faster than PCIe IIRC. Not ideal of course, but still better than a bunch of PCIe transfers.

Can you explain what do you mean by less useful ? and why ? (thank you)

The 970 is a cut-down 980. Where the 980 has 4GB of memory connected to the core through a 256-bit bus to the 8 32-bit GDDR5 memory controllers the 970 only has a 224-bit (aggregate) bus between the memory controllers and the L2 caches. The last eighth of the memory can only be accessed in a bank-switched fashion that blocks access to the primary 3.5GB bank. The primary bank can be accessed almost as fast as the 980's memory (striping across 7 chips instead of 8), but accesses to the last 0.5GB bank can't be striped across multiple chips. Those two effects together make it impossible to treat the 970 as having a full 4GB of general-purpose RAM.

to some extent, NVIDIA GPU and CUDA's success comes from the better tool chain and recently released cuDNN. Why AMD still hasn't release any alternative deep neural network library yet?

What about a NVIDIA Tesla which seems to be made for this kind of thing?

Tesla and GeForce products are made from the same chips. The GeForce products have several features turned off, and are then sold for a fraction of the price and often with more aggressive clock speeds. For any given workload, you have to assess whether you need the extra reliability of the Tesla products and whether the features disabled in the GeForce matter to the problem at hand. For many uses, the high-end consumer parts come out way ahead in this comparison; the Tesla parts are never an automatic win.

Tesla really shines when you want double precision computations, e.g., for scientific computing, solving PDEs etc. Extra precision is however seldom useful in deep learning, so you will likely get the same performance with the much cheaper GeForce series.

Reply on news.ycombinator.com