Intel AVX10: The Successor to AVX-512
phoronix.com
phoronix.com
This new ISA seems to include 256 bit versions of the instructions which were previously only in AVX-512 though. That's exciting.
The reason is that in Zen 4 the relatively narrow instruction decoder is frequently the bottleneck, so twice less instructions can influence the speed very much.
On future CPUs that will be able to decode more instructions in parallel, like Zen 5 and the future Intel CPUs, 512-bit instructions might no longer increase the speed when the width of the execution units remains of 256 bits, so perhaps even AMD may introduce a split between laptop CPUs that implement only the 256-bit subset of AVX-512 rebranded AVX10 and server CPUs that implement the full AVX-512 a.k.a. AVX10.
Single-pumped AVX2 implementations already had ~150 extra 256-bit registers for pipelining purposes. And if you're double pumping 512-bit instructions, then you can pipeline just as deep without changing the quantity or width.
Maintaining similar performance is not 512B->2KB, it's 5KB->7KB.
(But note that the E cores are largely double-pumping 256 bit instructions already, so for them it's a big change.)
"Intel AVX10 Version 1 will be introduced for early software enablement and supports the subset of all the Intel AVX-512 instruction set available as of future Intel Xeon processors with P-cores, codenamed Granite Rapids, that is forward compatible to Intel AVX10. This version will not include the new 256-bit vector instructions supporting embedded rounding or any of the new instructions and will serve as the transition base version from Intel AVX-512 to Intel AVX10.
Intel AVX10 Version 2 will include the 256-bit instruction forms supporting embedded rounding as well as a suite of new Intel AVX10 instructions covering new AI data types and conversions, data movement optimizations, and standards support. All new instructions will be supported at 128-, 256-, and 512-bit vector lengths with limited variances. All Intel AVX10 versions will implement the new versioning enumeration scheme."
And who knows when AMD will have time to update Zen ? architecture with these new instructions.
So AMD has plenty of time to implement the additional instructions, of which there don't appear to be that many. Mostly it's a unification of feature sets and a simplification of the CPU ID flags.
Also, Intel is still nerfing the efficiency "E" cores to AVX10-256, so the instruction sets will remain non-uniform in practice for the foreseeable future.
Meanwhile, AMD is implementing AVX-512 on all of their CPU cores including in the high-density 128-core AMD EPYC "Bergamo" line, which is a significant differentiator.
Intel just cannot resist segmenting a market, even when announcing that they're ending market segmentation!
Architectures take years to plan. If AMD is just finding out, I'm not sure they have time.
However, one would hope AMD had a little heads up. Its in Intel's and AMD's best interest not to flub the next AVX, with ARM SVE2 (and who knows what else) around the corner.
> being able to work for both P and E cores
Oh yes I forgot they were gimping their own processors.
> the converged version has a maximum vector length of 256-bits [on] the E cores while P cores will have optional 512-bit vector use
Maybe they shouldn't have made xmm and ymm "extensions" to the base set to begin with.
Guess Intel’s feeling the pressure from Zen 4 supporting AVX-512.
for (int i = 0; i < N; i++)
{
a[i] = b[i]*c[i];
}
New instruction sets usually do two things - add support for new operations, and widen the registers and allow you to perform the operation on more elements at a time.
If you’re writing in a high level language, you’ll only really see performance improvements if your interpreter or library takes advantage of them. For e.g. the Python library NumPy makes good use of vectorisation.
In terms of writing low level code, simple loops are often auto-vectorised by the compiler so you often see a sort of odd style where a loop doing many things is split up so the compiler can deal with it. You often end up having to run the code through something like VTune to work out whether a particular loop has actually vectorised.
The main benefit of AVX-512 is that the CPU gained support for "masking". Traditionally, you had to execute the instructions on all data elements. Something like "round all even values downwards, round all odd values upwards" becomes near-impossible, and if you want to do that in an entire AVX pipeline you have to deconstruct the vector into individual elements, do the operation on each of them individually, and reconstruct it into a vector. This really sucks. With masking, you get a special field which specify on which elements the operation should apply. So you could just create a mask with the even values, use that mask for a round-down, invert the mask, and use it for a round-up. That's a significant speedup!
Adding AVX-512 support means that a lot more applications suddenly become eligible for fairly trivial vectorization, and I personally can't wait for it to become universally available.
> As out-of-order CPUs continue to become deeper and wider, the cost of mispredictions increasingly dominates performance of such workloads. Branch predictor improvements can mitigate this to a limited extent only as data-dependent branches are fundamentally hard to predict.
> To address this growing performance issue, we significantly expand the conditional instruction set of x86, which was first introduced with the Intel® Pentium® Pro in the form of CMOV/SET instructions. These instructions are used quite extensively by today’s compilers, but they are too limited for broader use of if-conversion (a compiler optimization that replaces branches with conditional instructions).
> Intel® APX adds conditional forms of load, store, and compare/test instructions, and it also adds an option for the compiler to suppress the status flags writes of common instructions.
https://www.intel.com/content/www/us/en/developer/articles/t...
I didn't understand everything about the "caller-saved volatile" new general purpose register interface & legacy compatibility. But some potentially really interesting optimizations where load/store being dual register capable, and being capable of staying on the AVX unit & not having to go further out to "memory" (caches?):
> Generally, more register state will need to be managed at function boundaries. In order to reduce the associated overhead, we are adding PUSH2/POP2 instructions that transfer two register values within a single memory operation. The processor tracks these new instructions internally and fast-forwards register data between matching PUSH2 and POP2 instructions without going through memory.
Neat stuff. Very superficially reminds me of Semantic Streaming Registers on the very novel standalone-ish FPU on PULP's RISC-V based Occamy many-core chip. In that the unit is acting in a more standalone fashion. https://www.youtube.com/watch?v=kMhdq7A3d3I#t=10m https://pulp-platform.org/docs/BeniniSC11-22.pdf
Whenever new registers are added to a CPU, it may be necessary to modify the operating system in order to save the new registers across context switches, but it is possible to change the ABI (i.e. the conventions for calling functions) in such a way that will allow the new functions that use the new registers to be linked with old functions that are not aware of the existence of the new registers.
To achieve this goal, it is enough to define in the ABI all the new registers as "caller-saved volatile".
This means that any function that uses some of the new registers must not assume anything about the values of those registers upon function entry and it must save them before calling any other function.
This ensures that regardless what other functions do, intentionally or accidentally, the values stored in the new registers cannot be corrupted, so it is safe to mix new functions that are aware of them with old functions that ignore them.
Had the new registers been defined e.g. as registers that must be preserved across calls, i.e. which must be saved inside the invoked function, that would not have worked, because an old function would not have known how to save them.
Not every function boundary is an ABI boundary, and compilers can and will automatically use custom ABIs for static functions (and, with LTO, for unexported global functions, too).
Because the only thing I can pick out is the 256bit AVX-512 which AFIK recent amd processors do support (including 512bit support) both on their normal cores and their new compacted code.
But I don't know much about AVX_ so I'm 100% I missed a bunch of stuff and/or limitations with current AMD code (besides it being double pumped).
Half-precision floating-point for sure.
Sapphire Rapids has added an alternative format FP16, which is not supported by Zen 4, because Zen 4 was launched before Sapphire Rapids.
https://twitter.com/InstLatX64/status/1646471371558461445/
It's worth noting that 16-bits floating point operations are still possible on Zen 4, but using Google's BF16.
- AVX512ERAVX512PF (Knights Landing)
- AVX512 4VNNIW, 4FMAPS (Knights Mill)
- VP2INTERSECT (Tiger Lake)
- FP16 (Cooper Lake)
You can see a table here showing cpu's AVX512 support[0], and wikicpu also has that data in a list [1], along with more detailed info. It's worth noting all data I can find specifically talks about AVX512 support in terms of the architecture (Zen4), not anything specific product families such as ryzen vs epyc implementations. It's possible there is/will be some artificial product segmentation going on (like what Intel did retroactively with Alder Lake to disable AVX512) but I've not seen any mention of it regarding AMD's products. I mention this because Intel has kinda been all over the place with AVX512 implementation, and there's a big difference between Enterprise and Consumer in how they deliver/treat AVX512. AMD obviously is a different company with different practices, but with only 1 generation to go off of it's hard to speculate on a trend, and AVX512 can be an expensive implementation. Point being: It's unfortunately complicated AND subject to change.
The "big" one here for ML/DL/AI vs Intel's Sapphire Rapids (and presumably going forward) is FP16 not being supported on AMD. Of course, it's contextual as to whether that's relevant to you though.
[0]: https://en.wikipedia.org/wiki/AVX-512?oldformat=true#CPUs_wi...
[1]: https://en.wikichip.org/wiki/amd/microarchitectures/zen_4#Ne...
My experience is primarily with Intel's implementation of AVX512 in their Xeon products for DL/AI. I can expand on that if interested, but unfortunately haven't had a chance to play with AMD Zen4 yet in that domain, and it's gonna be a while till I can get hands on with a Epyc 9000 probably.
EDIT: Fixed typos
And yes, true. That list I just copied from wikichip, didn't think to cross verify what it was conveying in those paranthesis as the question was about AMD not Intel. That said, not sure why they have FP16 attributed to Cooper Lake and not SR.
EDIT: Ayy, you can edit wikichip. Fixed it over there (assuming there's not some moderation issue? never submitted anything over there), but too late to edit original HN Comment.
AMD practically resurrected AVX-512 by supporting it across both desktop and server products, so it's weird to criticize them for taking a while when Intel nearly killed the thing via segmentation and poor implementation.
AVX512 vector and mask register take space on the die, which is a finite resource and could be allocated to other things.
Moreover, power consumption may have also been a challenge, as seen in the early Intel implementations of AVX512 (especially Skylake-X).
And while AVX512 can improve single-threaded performance, that's only for applications that make use of it. And up until recently, that number was extremely limited.
But who knows if AMD sees value in these new instructions/if there's market demand for them.
AMD Zen4 has 256-bit load/store to L1 cache, which means it'd take a load/store unit 2-clock ticks to execute any 512-bit load or store. (Though Zen4 has multiple load/store units, so maybe its not that big of a deal in practice).
-------------
GPUs are 32-wide with 32-bits each, or 1024-wide operations in practice. GCN was 64-wide with 32-bits each or 2048 wide in practice.
The reason why AVX512 was a big deal isn't the width btw. But instead the design of the instruction set. AVX512 is the most advanced CPU vector instruction set ever made, with huge inspirations from NVidia's PTX or AMD's GCN.
Of course, GPU-assembly remains superior. I don't think that RISC-V (or whoever) should be copying AVX512 or even AVX10, they should be looking at NVidia PTX or NVidia SASS as inspiration.
Can you explain what makes it the above? I haven't really worked with avx512 before, only up to avx2 and rvv.
2. AVX512 has vgather and vscatter, aka vector-load and vector-store. While no computer accelerates this process, this still simplifies programming and read/writes to memory can be thought of on a per-lane basis.
3. Compress and Expand, new instructions not found in GPU-space, perform the stream-compression operation (and inverse operation). See: https://www.cse.chalmers.se/~uffe/streamcompaction.pdf , lots of practical applications here, nice to see just a simple "instruction" that handles this detail. Though GPU programmers can prefix-sum and build this primitive themselves in practice, its a common enough operation that a proper-instruction is likely a good idea.
------------
Just a few tidbits of AVX512 ISA that shows that Intel really knows what they're doing here.
#1 can be seen as a 64-bit generalization to the cmov conditional instructions that are in Intel's normal 64-bit instruction set. Except instead of being a "1-bit" compare or 1-bit if-statement, might as well generalize it to a 64-bit operation across 64x8-bit bytes that naturally map into a 512-bit register, right?
Its "why" you can just say "if(blahblah)" in CUDA or OpenCL, because under-the-hood, the GPUs are holding onto these 64-bit conditional registers and are turning-off and turning-on lanes to execute. (GPUs don't do if(){A();} else () {B();} like a CPU. GPUs turn on conditional-execution masks per-lane and then execute both A-and-B, though if the lane is "off" then the results are thrown away). It all comes down to S-registers (GCN), opcode-masks (AVX512) or whatnot, its fundamental to "proper" GPU-like SIMD.
In any case, AVX512 is still playing catchup to the gpu space, but Intel has shown me (IMO) that they're the CPU-assembly group that has fully understood what makes GPUs tick and why parallelism is possible in CUDA / OpenCL. Maybe they'll leapfrog the GPU guys in the next few years, but I still consider GPUs to be the cream-of-the-crop in terms of assembly-language design for parallel SIMD systems... for today anyway.
Then again, NVidia has also shown themselves to be very innovative in this low-level space. And in many ways, its easier to "catch up" (as Intel has done) rather than trailblaze. Still, Intel showing off compression instructions is a good sign, it is true innovation.