timdettmers

Thank you so much!

Haha, 649 pages! Thanks for the discussion. I can understand your perspective. Maybe it would be best to add something to the blog post that discusses it from both perspectives and readers can then choose which perspective suits them better.

I should also a bit more in general about cloud computing, it seems some people agree that the post ran a bit short on that. At some point I just wanted to be done with it though — editing 10k word blog posts is not so much fun anymore!

Thank you! You have a good point. I think I would agree with you if somebody already has cloud computing skills, then the cloud is much more powerful to learn deep learning than your own GPU.

I figured that most people that start with deep learning might also lack cloud computing skills. Learning one thing at a time is easier and as such, just sticking a GPU into your desktop and focus on deep learning software / programing might yield a better experience.

I might update my blog post in the future with this detail.

I just did a small fix to address this. Thanks again.

Thank you, I just updated the blog post with more detailed clarification of where the data comes from.

One thing that I am quite sure of for the A100 is its transformer performance. It turns out, large transformers are so strongly bottlenecked by memory bandwidth that you can just use memory bandwidth alone to measure performance — even across GPU architectures. The error between Volta and Turning with a pure bandwidth model is less than 5%. The NVIDIA transformer A100 benchmark data shows similar scaling. So I am pretty confident on the transformer numbers.

The computer vision numbers are more dependent on the network and it is difficult to generalize across all CNNs. For example, group convolution or depth-wise separable convolution based CNNs do not scale well with better GPUs and speedups will be small (1.2 - 1.5x) whereas some other networks like ResNet get pretty straightforward improvements (1.6x-1.7x). So CNN values are less straightforward because there is more diversity between CNNs compared to transformers.

I should have been a bit clearer what went into the charts. I do not use theoretical marketing numbers but real-life benchmark data from NVIDIA and 4 other sources of benchmark between Titan V, V100, RTX 2080, RTX 2080 Ti and Titan RTX. Since I calibrate a model that needs to satisfy all sources as best as it can I think the numbers are pretty accurate.

This is good feedback. Will note this down and incorporate it in a small update.

What else would you like to see?

Is up again and stable it seems. I shot my cache while trying to "tune" it. Well, I learned my lesson.

Please see my detailed answer on this question that you can find on quora: https://www.quora.com/Why-are-GPUs-well-suited-to-deep-learn...

Your data are inconsistent with the benchmarks that I mention in the blog post: https://github.com/u39kun/deep-learning-benchmark

You also do not benchmark LSTMs: https://www.xcelerit.com/computing-benchmarks/insights/bench...

If you put both of those benchmarks together my conclusion is quite reasonable. But I see that you could also come to your conclusion with your benchmarks. It is just a question which benchmarks are less biased and that is too difficult to evaluate.

I guess we have to wait for real data, but thanks for putting your data out there to get a discussion going.

Thanks for your feedback! Someone mentioned this on twitter as well and I thought it was a good point so I implemented that change.

The site is up again.

I have a prepared draft for a blog post exactly on the topic of cloud computing and deep learning. I did not finish it as I thought that there would not be much interest in the overall question since most people will just buy GTX cards. However, it seems that there is quite a confusion going on what makes sense and what does given certain circumstances. I think I will finish that blog post now and post it in the next days.

If you want me to discuss certain questions regarding deep learning hardware and cloud computing let me know here.

A very valuable comment – this is an important perspective, thanks! I think I will add a EC2 section to my blog post.

Thanks for sharing your experience – this is a fair point. Often it is possible to pre-process your data and save it to disk so that you can skip this decompression/conversion/transformation step once you start training your net, but I can image applications where this is impractical or just does not work. I will add a small note to my blog about this.

Thanks four your feedback, this is an important point and I will update my blog posts accordingly.

These are some good points. I heard from another person that he had problems with the form factor and I will add that to the post tomorrow. I think a 2kW PSU is overkill, but you are right that more is better for PSUs.

If you want memory a good option will be to wait for the GTX Titan X which will be released in the next weeks: 12 GB RAM and it will be the fastest card by far. Overall however, I think the GTX 980 will be better for many cases still – it is just very cost effective.

If you have no desktop PC or no money for a GPU, it might be a better choice to use a EC2 instance instead of buying the hardware. You pay about $11 a week for a EC2, which is quite good once you compare it against the electricity costs that come on top of running a personal computer.

The downside is that you have a slow EC2 GPU with 4 GB RAM. Conv nets that take 3 weeks on a EC2, will take less than 2 weeks on a GTX980. If you run large conv nets, the 4 GB can be limiting (for example on ImageNet or similarly sized data sets).

Another point is that is more convenient to work on your own desktop and you can run multi-GPU nets, which is not possible on EC2 because the virtualization kills the memory bandwidth between GPUs.

If you think about it, over the long term a personal system will just be more cost efficient (you can keep a good system for years). So for deep learning researcher and those that apply deep learning this is just the most cost effective option.

A example calculation: You can buy a faster system than a EC2 for roughly $400 (GTX 580 + other parts from eBay). Together with electricity costs thats about 1 year worth of EC2, or 2 years worth if you use deep learning sporadically. A high end deep learning system will be about $1000-1400, which is about 3 years worth of EC2. So a EC2 makes good sense, if you use deep learning only sporadically and work with small data sets. If you use deep learning heavily, want a faster system or want to use multiple GPUs, a personal system will be better.

There are several cryptocurrency mining benchmarks which are bandwidth bound; those benchmarks reflect almost one to one the deep learning performance of a GPU. You can see this for example for litecoin mining: https://litecoin.info/Mining_hardware_comparison

Thanks for your feedback. I updated the blog post with a small NVIDIA vs. AMD section.

Thanks for elaborating, these are exactly the points why AMD's GPUs are just not used in deep learning.

AWS is great if you want to use a single or two separate GPUs. However, you cannot use them for multi-GPU computation as the virtualization cripples the PCIe bandwidth; there are rather complicated hacks that improve the bandwidth, but it is still bad. Everything beyond two GPUs will not work on AWS because their interconnect is way to slow.

If you are dissatisfied with the 970 you might be able to get a refund: NVIDIA advertised specs that differ from the real thing – that is a good reason to demand and get a refund.

Tesla GPUs are also rather slow, but the value of Tesla lies in increased double performance, larger memory and memory correction -- and you cannot beat Tesla GPUs in these departments.

AMD cards were much faster than the Kepler equivalents, but now NVIDIA fixed their mistakes in that architecture and they are as fast as AMD cards. So AMDs hardware is good, it is just a software issue.

Why I did not mention AMD cards: NVIDIA's standard libraries made it very easy to establish the first deep learning libraries in CUDA, while there are no such powerful standard libraries for AMD's OpenCL. Right now, there are just no deep learning libraries for AMD cards -- so you just cannot use AMD cards for deep learning.

I mainly use python and sklearn for Kaggle competitions for my initial models. If I understand the problem better I use some of my own deep learning solutions in python (built on gnumpy and cudamat). However, sometimes my own C++/CUDA implementations come in handy, especially if the data set is large.

Other Kaggle competitors that use deep learning mostly use python libraries like pylearn2 and torch7 for their deep learning models (which are also built on CUDA/C++).

In general it is not so easy to use deep learning on problems other than object recognition. So yes, I do not use deep learning in all of my Kaggle competition simply because it is hard to get them to work well. Using different simple models and to then ensemble them yields often better results for the time invested.