Is there a description somewhere without all the bla bla hype? Comparison with past architectures would also be welcome, also without hype.
So, yeah, those tables are hype too. I am asking about benchmarks on real workloads.
Moving from a GTX 480 to a Kepler or Maxwell card, the numbers go up, but not the performance. I might have a corner case, but before investing in new hardware, I would want to benchmark first and not blindly follow the numbers.
* ~5.5 TFLOPs on FP64
* "About 2x" performance on FP32 (so 11 TFLOPs)
* "Up to 2x" performance on FP16 (compared to FP32, so about 22 TFLOPs).
* FP16 is also aimed to neural nets training, because when the weights of the net are FP16, it's a more compact representation.
* 3840 general purpose processors.
* More/better texture units, memory units, etc. So it's not about raw power, but also about a better design.
Guess that's about the important stuff. I just skimmed the article over the top, reading a bit here and there, but that seemed to be the most remarkable stuff.
For historical performance, just pick one of the machines that did x teraflops. E.g. the first teraflop computer used ~6000 200MHz pentium pro chips around 1996.
In describing the benchmark, they say,
In an attempt to obtain uniformity across all computers in performance reporting, the algorithm used in solving the system of equations in the benchmark procedure must conform to LU factorization with partial pivoting. In particular, the operation count for the algorithm must be 2/3 n^3 + O(n^2) double precision floating point operations. This excludes the use of a fast matrix multiply algorithm like "Strassen's Method" or algorithms which compute a solution in a precision lower than full precision (64 bit floating point arithmetic) and refine the solution using an iterative approach.
So, to summarize, if in 2000 the fastest supercomputer on the planet ran at about 4.9 TFLOPs, does that mean, apples-apples on the LINPACK (and only the LINPACK), that Pascal today would outperform that Supercomputer?
The benchmark is essentially bottlenecked on FP64 matrix multiplies. If that's what you need to do, then sure, it's indicative.
Some machine learning workloads are also bottlenecked on matrix multiply, but don't need FP64 precision. They can use FP16. Fits a bigger model in a given memory size, makes better use of memory bandwidth, and given the right hardware support, you can get extremely high rates as on Pascal.
Personally, I find the memory system on Pascal more interesting than raw flops rate. Also, the use of nvlink to link multiple GPUs..
There is a hierarchy of HPC-type workloads called "Colella's seven dwarves" that ranks different workloads in terms of being CPU bound or memory bandwidth bound. See also the "roofline model". Both of these heuristics are made to reason about CPUs, but are also effective for thinking about GPUs.
People/manufacturers tend to look at clock rates, fill rates (for GPUs), FLOPs, "crunching power" in general, forgetting completely the memory part. For example, today most CPUs end up being bound by cache sizes and performance tuning focuses on being nice on the cache rather than being optimal in your instructions (see for example Abrash's Pixomatic articles[0-2], which are about high performance assembly programming in "modern environments").
With GPU and "classic" HPC (don't know about the new systems with the "compute fabric interconnects"), memory usually becomes the bottleneck (except for embarrasingly paralell problems, of course). In fact, I'm pretty it was Cray who said that a supercomputer is a way to turn a CPU-bound problem into an IO-bound problem.
[0] http://www.drdobbs.com/architecture-and-design/optimizing-pi...
[1] http://www.drdobbs.com/optimizing-pixomatic-for-modern-x86-p...
[2] http://www.drdobbs.com/optimizing-pixomatic-for-modern-x86-p...
What nvidia calls a "core" is more like one entry in a SIMD unit on a cpu. What nvidia calls a "SM" is closer to a CPU core.
There is more to it than that, i.e. gpu cores are more independent than entries in a cpu vector unit, but on the other hand, but gpu "SM"s are less independent than cpu cores.
It's also worth keeping in mind that mediocre cpu code will run circles around mediocre gpu code. To get the gpu magic you have to invest a lot of effort in tuning for the architecture.