Your assertion sounds like a gross exaggeration. Some benchmarks show only a few percent improvements (i.e.14% improvements on synthetic benchmarks), and results range between no improvements and almost 2x in some synthetic benchmarks.
https://www.phoronix.com/scan.php?page=article&item=gcc9-sky...
To be clear I am not suggest "avx-512 will cause programs to run twice as fast or more", just that specific functions that used to use AVX-2, may be more than twice as fast if you move them to AVX-512.
I presented data that showed that AVX-512 in reality only brings margins performance improvements.
If you have any data that shows otherwise then feel free to present it.
Still, hand-tuned assembly is hardly a relevant example, let alone selling point, as it creates hypothetical expectations that are never ever matched in reality.
There's a ton of hand tuned assembly running on your system in numerous libraries. Media and data compression? Hand tuned assembly. Matrix math? You guessed it. C-standard library functions? Yes, really.
Certainly you have required architecture instructions for things like instruction scheduling (isync/dsync), stack manipulation (setjump/longjump and others) or floating point, but though it looks like a lot of assembly source, it is really just a little bit for each of the many supported architectures in the tree. And most of it isn't performance related, it is just plain required because architectures vary on things like instruction scheduling and stack/cache manipulation.
The only AVX512 instructions that actually seem essential imo are the VNNI extensions.
There are a lot of use cases for a neural network that is trained on the GPU but where inference is done on a bunch of CPUs because it's cheaper than buying/renting inference hardware.
There's a reason AMD is and Intel are able to cram full GPUs onto the same die as the CPU.
Yeah, here's a good image of how much space it costs to add a 'second' AVX-512 unit to certain SKUs. Cutting AVX-512 support entirely would save a bit more space, and the total would be enough to make a difference, but it's not an extreme amount.
https://software.intel.com/sites/default/files/managed/5f/99... https://twitter.com/GPUsAreMagic/status/1256866465577394181 these too (speculative)
With AVX-512 and CUDA, Intel and Nvidia are trying to find the best developer experience they can to deal with the inevitable future of computing that is somewhere between those two experiences currently.
Not sure that ever happens. The two are very different in their performance characteristics. CPUs have to stay focused on single-threaded performance.
Single threaded performance = minimizing latency, that’s why in CPUs so many transistors and electricity is spent on cache hierarchy and coherency protocol, instructions reordering, ILP, microops fusion, prefetcher, branch prediction incl. indirect one, and so on.
GPUs don’t care about single-threaded performance, all they care is throughput on massively parallel tasks. They ignore memory latency, switching threads until data arrives, makes caches much simpler. They don’t need high clock frequencies (this alone improves their flops/watt by a large factor), don’t do branch prediction, memory model is less strict, and so on. That’s how they can spend that many transistors on FPUs and ALUs.
P.S. Intel tried many-slow-cores with Larrabee / knights landing / etc., all they got was a very expensive lesson, people didn’t want that stuff. I’ve heard a rumor initially Sony wanted to use another Cell CPU for the GPU in PS3, again, didn’t worked well enough and they negotiated with NVidia instead.
We can still add more transistors. But, there is nowhere for them to contribute but in going wider. High latency, high throughput. High concurrency and careful management of the memory hierarchy are the only ways to pull that off. When you accept that fate, you start out with the Cell processor and you pass through Nvidia's Ampere chip --which has somewhat more thread independence and better automatic caching than past GPUs, but also finally gained manual async DMAs to local SRAM like the Cell had. Where we'll end up, I don't know.
I agree on physical limitations, but the research still goes on, smart people still able to use these extra transistors in a way that’s useful for performance. Even for single-threaded one.
Another thing, software is progressively better at using many cores. It’s much easier to move large pieces of code to another CPU thread (like some browsers do with JS compiler) keeping it sequential code, than it is to re-write code as a large set of small tasks / threads. Many useful algorithms can’t use fine-grained parallelism much, or at all, e.g. streaming stuff like gzip or parsers.
https://www.karlrupp.net/wp-content/uploads/2018/02/42-years...
BTW: Thanks for http://const.me/articles/simd/simd.pdf I came across it recently and posted it to https://www.reddit.com/r/simd/
Someone should probably tell AMD and Intel they need to stop, then, because Ice Lake was +18% IPC, Tiger Lake will likely be another 5-10%, and Zen3 will be around 15% higher IPC.
What has truly stalled IPC is 10nm. Intel has had higher-IPC architectures for a long time, they simply couldn't manufacture them. And AMD was coming so far from behind that it took them 3 generations of Zen to even catch up to Intel.
The last few years have been a rather unfortunate "CPU dark age" but things are finally breaking loose once again. Ice Lake launched 9 months ago, Intel backported Tiger Lake to 14nm and that will launch probably around Christmas or CES, Milan is launching probably within the next 6-8 weeks, and Tiger Lake is launching this month.
CUDA in particular hides the exact width of the processing elements from the programmer, so you don't have to rewrite your CUDA applications to take advantage of the next generation of GPU the way you have to do with AVX1,2,3,...
Mind you, NVIDIA's drivers compile your shaders to their proprietary instruction set, which is a path not taken in conventional CPU architecture even though it keeps coming up. (e.g. the belt)
In enterprise ML contexts on tabular data problems we found there are a lot of cases where even training can be greatly sped up by leveraging AVX instruction support in e.g. tensorflow builds. The gains from AVX instructions could boost training time by ~20% on the GAN use cases I profiled.
What percentage of consumers with these chips installed do you think are getting a performance win? Do you think it might be as high as 1%?
Do you think that the same resources devoted elsewhere might be worth more than 1% to that 99%? Whether that is in reduced cost, reduced bugs, or a boost for more widely used operations.
Yes, if you target a specific use case for a specific set of people, you can give them a nice win. But you shouldn't lose sight of the fact that CPUs cover a lot of use cases for a lot of people. And simplifying then focusing on the core mission is better for everyone in the end.
Turning factual reports of where the tradeoff was helpful into a strawman insulting the reporter, and the users who benefit, is neither charitable nor illuminating.
If your algorithm isn't sensitive to either one of these, then yes, the speedups are often smaller, particularly if the limiting factor in performance is memory bandwidth and/or latency (AVX-512 doesn't help at all there), or if the speedups aren't enough to overcome clock throttling (which IMO is a bit overblown; most people cite the Cloudflare blog article from a few years ago that complained about throttling on Xeon Silver processors, but throttling is much less of an issue on higher-TDP Gold/Platinum CPUs). However, I have seen specific applications that are >100% faster with AVX-512.
Current AVX512 implementations might not always get there, as I believe some instructions are dual issuing at 256bit under the hood.
Ultimately AVX will be the future of Intel if they want to compute with GPUs and if everyone stays silicon.
Eventually, I see us going basically full EPIC. That is "here's a batch of instructions that can happen in parallel, run it".