AVX512/VBMI2: A Programmer’s Perspective
singlestore.com
singlestore.com
The regularity and simplicity and completeness of the instruction set is a big win.
The lane predication (masking) of every instruction is useful when the data doesn't fit the vectors (and in loop tails) but it has numerous other uses, too. For example, it makes blending (vector splicing) and partial operations easy.
The flexible permute instructions (two data vectors in, one control vector in, and one data vector out) are fast and enormously useful. Anyone who has puzzled with AVX2 will breathe a sigh of relief.
The register file is big enough (32 vectors, each 16 floats wide) that register starvation typically isn't an issue. For example, you don't worry about having registers for your permutation control vectors. And the predicate masks are in another set of registers, which again are plentiful (eight!).
The easing of alignment requirements (unaligned load/store instructions operating on aligned addresses are as fast as aligned loads/stores) is also a big win.
AVX-512 is a real pleasure to use.
Arbitrary swizzles both in the forward permute (currently in AVX512) and backwards direction (GPU only) is as necessary and proper as gather + scatter. Any program that uses one is highly likely to use the other.
--------
Butterfly permutes should be especially accelerated, as that pattern continues to show up. It seems like arbitrary permutes / bpermutes are expensive to implement in hardware, but butterfly permutes are the fundamental building block.
Butterfly permutes are needed from FFT to scan/prefix sum operations. It's also fundamental (and simpler at the hardware level than arbitrary bpermute/permute)
AMD implements the butterfly permute in DPP instructions. NVidia provides a simple shfl.bfly PTX instruction.
Butterfly networks (and inverse butterfly) are how pdep and pext are implemented under the hood. http://palms.ee.princeton.edu/PALMSopen/hilewitz06FastBitCom...
-----------
IIRC,the butterfly / inverse butterfly network can implement any arbitrary permute in just log2(n) steps.
As best I can tell, it would be reasonable to expect around 12 AVX2 cores operating on the same power budget as the current 8 AVX512 cores. Does the average user have enough AVX512 workloads to make that tradeoff worth it?
Any media processing can generally gain significant performance with SIMD instructions. Even web browsing is a workload that is affected, as (among other things) libjpeg-turbo uses SIMD instructions to accelerate decoding. It doesn't yet use AVX512, but it does use AVX2. If there are gains with AVX2, why wouldn't there be with AVX512.
Even if AVX512 would remain 256 bits, it would be an improvement over AVX2 as new instructions enable acceleration of workload that were not possible before, plus there is an increase in productivity/ease of use.
However, AVX512 won't be used much for consumer applications until there is big enough market penetration of CPUs that support these instructions, as it has been the case with any new SIMD instructions when they were first introduced.
To a limit. JPEG (and many video codecs based on JPEG) have 8x8 macroblocks, which means the "easiest" SIMD-parallel is 64-way. And AVX512 taken 8-bits at a time is in fact, 64-way SIMD.
To get further parallel processing after that, you'll probably have to change the format. GPUs go up to 1024-way NVidia blocks (or AMD Thread groups), which are basically SIMD-units ganged together so that thread-barrier instructions can keep them in sync better. 1024-work items corresponds to a 32x32 pixel working area.
But that's no longer the format of JPEG. It'd have to be some future codec. Maybe modern codecs are seeing the writing on the wall and are increasing macroblock size for better parallel processing 10 years into the future (they are a surprisingly forward looking group in general).
We did indeed do this for JPEG XL - the future is now :) 256x256 pixel groups are independently decodable (multi-core), each with >= 64-item (float) SIMD.
Cache lines are probably 64 wide for the purpose of burst length 8 (64 bit burst length 8 is 64 bytes / 512 bits).
There was AMD's 3DNow! that saw limited software adoption, because Intel didn't support it. Newer instruction sets are getting adopted progressively slower as consumers are replacing computer less often, AMD is slower at adopting each new AVX instruction set and Intel is getting more aggressive with market segmentation. Because market penetration of new instruction sets is getting slower, SW adoption is also much slower.
Not even all gamer CPUs have SSE4 (https://store.steampowered.com/hwsurvey), so it seems that runtime dispatch is unavoidable.
Given that, if we can afford to generate code for new instruction sets and bundle it all into one slightly larger binary/library, the problem is solved, right?
Highway makes it much easier to do that - no need to rewrite code for each new instruction set. As to Intel's market segmentation, we target 'clusters' of features, e.g. Haswell-like (AVX2, BMI2); Skylake (AVX-512 F/BW/DQ/VL), and Icelake (VNNI, VBMI2, VAES etc) instead of all the possible combinations.
It's less than 2x, because the core downclocking for AVX-512 on older CPUs is higher than for AVX2: 60% vs 85% on Skylake, so only a ~1.4x speedup. Newer CPU architectures do not downclock, though.
Newer CPU architectures may not have enforced downclocks, but they will eventually respond to the higher thermal output.
[1] https://en.wikichip.org/wiki/intel/microarchitectures/sunny_...
[2] https://www.realworldtech.com/forum/?threadid=193291&curpost...
That analysis might be slightly underestimating the area penalty of AVX512, because the consumer Skylake cores that didn't have AVX512 execution units still reserved space for the AVX512 register file. (And given that fact, it's all the more surprising that while Intel was repeatedly refreshing 14nm Skylake for the consumer market, they never added the rest of the AVX512 bits or redid the layout of the consumer cores to reclaim the blank space of the register file.)
If you're going to hypothesize about a rearranged Skylake core that no longer reserves space for the full 512 bit vectors in the register file, then you probably should also try to estimate the other area savings that would result from truly removing AVX512 from the core design at a high level, rather than merely masking off a few regions of silicon that serve no purpose other than AVX512 support. And answering that question is a lot more subjective than simply tallying up the area occupied by the extra execution units hanging off the side of the AVX512-enabled server Skylake cores.
Or, to put it another way: there unquestionably is an area cost to AVX512 beyond that of the extra execution units tacked on to SKL-X cores. But that cost was already being paid by consumer CPUs years before SKL-X/SKL-SP shipped. So it wasn't really a contributing factor to the loss of two cores when Skylaked-derived Comet Lake was succeeded by Rocket Lake.
Its seems most new CPUs support a fairly large subset of the older ones.
I haven't dont low lever work of a while but remember doing some analysis of a signal processing code going from PA-RISC to Intel. The math compiler libraries for PA-RISC made the initial code transition so much slower on Intel. Using the Intel compiler and tweaking things started making things work so much better.
One hopes that ARM's ascendance would make "team x86-64" work in cooperative competition for better performance through compilers (it they can't through silicon).
(IIRC Intel has tuned down the Cripple AMD thing in MKL around late 2020 by providing a specialized Zen code path. It was slower than what you get with the detection backdoor, but only slightly.)
And then there’s the genius stuff like SIMD UTF8 decode and whatnot. Compilers don’t magically figure those out. Heck, they sometimes have trouble with shuffles, which is why the author mentions that the new features will help.
^1 Oh, not GCC on -O2 IIRC. There were some talks about turning autovec on that level on like clang does, but I’m not sure it went anywhere… I also think MSVC doesn’t do that.
^2 This pragma also turns on autovec even if the compiler is not told to do that with the flags. Although my original point was about the extra information you can provide with it.
But pretty much every JSON/XML/Number.from_string is using an operation like this, to the scale is quite large.
There is some opportunity for ISA support to speed it up, but multiplication by powers of ten is not the bottleneck, that’s just a table lookup and full-width product in high-performance implementations (i.e. a load and a mul instruction).
You cannot represent 10^x, or even all 5^x beyond a certain point in IEEE754 double's but need to do the full y * 10^x operation without loss.
Doing it lossy is easy(0 to 1 or 2 ulp off optimal)
This means you may need arbitrary-precision for arbitrary-precision decimal strings, but in practice these libraries are “always” converting strings that came from formatting exact doubles (eg round-tripping through JSON), and so have bounded precision. Thus you can tightly bound the precision required.
This precisely mirrors how formatting fp numbers used to require bignum arithmetic, but all the recent algorithms (Ryu, Dragonbox, SwiftDtoa) do it with fixed int128 arithmetic and deliver always correct results. We’ll see analogous algorithms for the other direction adopted in the next few years—the only reason this direction came later is that the other was a bigger performance problem originally.
Also, x87 has/had the FBLD and FBSTP instructions that can be used to convert between floating point and packed decimal (https://en.wikipedia.org/wiki/Intel_BCD_opcode)
If these still are supported, I doubt Intel or AMD spend lots of effort making/keeping them fast, though.
In short, you need to measure the combination of compiler options that get you the best performance on your real platform. Most people can probably pick up a quick 10-20% win by recompiling their MySQL or whatever for arch=sandybridge, instead of k8-generic, but beyond that it gets trickier.
Are you thinking of -mtune? I'd never heard of people using -march for performance, I thought it was just specifying the ISA.
In the particular case as I recall it was AVX2 that was counterproductive on haswell. Disabling it made the program faster, even though AVX2 was supposed to be a headline feature of that generation.
By far the simplest way of making the correct data-layout is to use the intrinsics manually, so that you discover any issues that arise.
A step further is to change the program logic to better take advantage of SIMD-instructions, for instance Fabian Giesen has written a great deal on making compression code that splices the jobs in creative ways in order to utilize SIMD-instructions.
I'm a programmer too but I don't share that perspective.
In my line of work, AVX512 is useless. This won't change until at least 25% market penetration on clients, which may or may not happen.
Not all programmers are writing server code. Also, very small count of them are Intel partners with early access to the chips.
> In AVX terminology, intra-lane movement is called a `shuffle` while inter-lane movement is called a `permute`
I don't believe that's true. _mm_shuffle_ps( x, x, imm ) and _mm_permute_ps( x, imm ) are 100% equivalent. Also, _mm256_permute_ps permutes within lanes, _mm256_permutevar8x32_ps permutes across lanes.
I'm not sure AVX512 is a win even in that case. AMD Epyc peaks at 64 cores/socket, Intel Xeon at 40 cores/socket. It's not immediately obvious AMD's performance advantage over Intel is smaller than the Intel-only win from AVX512.
The Xeon 40 core is just one die, which means L3 cache and memory is more unified.
L3 cache is probably king (data may not fit, but maybe some indexes?), but it's not obvious if EPYCs separate L3 caches are comparable to Xeons (or Power)
AMD Zen3 has single cycle pext / pdep now and the chess benchmarks are better as a result.
Still, this doesn’t mean Epyc is only good for HPC, e.g. this is some enterprise Java-based benchmark where it’s substantially faster: https://www.anandtech.com/show/16594/intel-3rd-gen-xeon-scal...
BTW, on my job I don’t do databases, but I do quite a lot of HPC stuff.
x265 is more of an AVX512 test scenario, which the Xeon 40-core also is demonstrating proficiency at over and above the EPYC 64 core.
------------
I'm mostly impressed at how well the split "chiplet" strategy is doing in all the other benchmarks however. The AMD "I/O die" is clearly a winner. Its probably not a latency winner, but it is efficiently giving the memory bandwidth and distributing it to all of the dies.
Why not use it where available? github.com/google/highway lets you write code once using 'platform-independent intrinsics', and generate AVX2/AVX-512/NEON/SVE into a single binary; dynamic dispatch runs the best available codepath for the current CPU.
Disclosure: I am the main author. Happy to discuss.
AFAIK things like that (highway, OpenMP 4.0+, automatic vectorizers) are only good for vertical operations.
When the only thing you’re doing is billions of vertical operations, computing that thing on CPU is often a poor choice because GPUs are way faster at that, and way more power efficient too.
Therefore, when I optimize CPU-running things, that code is usually not a good fit for GPUs. Sometimes the data size is too small (PCIe latency gonna eat all the profit from GPU), sometimes too many branches, but most typical reason is horizontal operations.
An example is multiplying 6x24 by 24x1 matrices. A good approach is a loop with 12 iterations, in the loop body 2x _mm256_broadcast_sd, then _mm256_blend_pd (these 3 instructions are making 3 vectors [v1,v1,v1,v1], [v1,v1,v2,v2], [v2,v2,v2,v2] ), then 3x _mm256_fmadd_pd with memory source operands. Then after the loop add 12 values into 6 with shuffles and vector adds.
I agree autovec and #pragma omp simd struggle with any kind of shuffles, but Highway does not belong in that category - it offers more control, similar to intrinsics.
There are about two dozen non-vertical ops (see https://github.com/google/highway/blob/master/g3doc/quick_re... and "blockwise" and "swizzle") and it sounds like you want Set and ConcatUpperLower.
In your experience, have you needed any non-vertical ops that are not in that list?
Yes indeed. I rarely using SIMD for vertical-only ops, for such use cases GPUs are very often better than CPUs.
I’ve already wrote an example in my previous comment. It’s possible to emulate with the stuff you have, however _mm256_blend_pd is very fast instruction, a single cycle of latency. The highway’s emulation is going to be way more expensive. You probably compiling your UpperHalf() into _mm256_extractf128_pd and Combine() into _mm256_insertf128_pd, that’s 2 instructions and (on Skylake) 6 cycles of latency instead of 1 cycle.
6 cycles instead of 1 cycle is a large overhead in that context. That particular small matrix multiplication is called rather often. I only optimizing code when the profiler tells me so. For the majority of CPU bound code in that project, Eigen’s implementation is actually good enough.
I’ve searched the source code of that project (CAM/CAE software). Here’s the list of the shuffle intrinsics I use, some of them a lot: _mm256_blend_pd, _mm_blend_ps, _mm_blend_epi32, _mm256_permute2f128_pd, _mm256_permute_ps, _mm256_permute4x64_pd, _mm256_permutevar8x32_ps, _mm256_permutevar8x32_epi32, _mm_permute_ps, _mm_permute_pd, _mm_insert_ps, _mm_movehdup_ps, _mm_moveldup_ps, _mm_loaddup_pd, _mm_extract_ps, _mm_dp_ps, _mm_extract_epi32, _mm_extract_epi64, _mm_shuffle_epi32.
A similar list for this project https://github.com/Const-me/Vrmac (a GPU-centric library for 3D and 2D graphics, not using any AVX): _mm_shuffle_epi8, _mm_alignr_epi8, _mm_shuffle_epi32, _mm_shuffle_ps, _mm_addsub_ps (that one is vertical but still missing from highway), _mm_insert_epi32, _mm_insert_ps, _mm_extract_ps, _mm_extract_epi16, _mm_movehdup_ps, _mm_dp_ps. BTW the project is portable between AMD64 and ARMv7, I have tons of #ifdef there to support NEON which differs substantially, there’s stuff like vrev64q_f32 and vextq_f32, 64-bit SIMD vectors, and quite a few other instructions missing on AMD64.
Even if you expose all the missing horizontal stuff in highway — won’t be much better than intrinsics. Such code ain’t gonna use AVX512 when available. Only going to inflate the software complexity for no good reason, by adding an unneeded layer of abstraction between the application’s code and the actual hardware.
Actually it uses exactly the same instruction :) https://github.com/google/highway/blob/master/hwy/ops/x86_25...
Thanks for sharing the list! Out of curiosity, can you expand on what _mm256_permute4x64_pd is used for?
> _mm_addsub_ps (that one is vertical but still missing from highway)
Indeed, we have not needed complex-valued functions yet. AFAIK NEON does not have such an instruction but it could be emulated with an extra XOR+constant. Will add to wishlist, we implement them whenever there's a use case.
> I have tons of #ifdef there
Seems reasonable if you only target v7 and ASIMD; but #ifdef is increasingly infeasible now that SVE and Risc-V V are coming, no?
> to support NEON which differs substantially, there’s stuff like vrev64q_f32 and vextq_f32
Those can actually be bridged just fine, they correspond to Shuffle2301 and CombineShiftRightLanes.
> Even if you expose all the missing horizontal stuff in highway — won’t be much better than intrinsics. Such code ain’t gonna use AVX512 when available.
Because of the hardcoded vector size? I agree that's best avoided, in your case perhaps by batching the matrix mul and applying SIMD over that dimension instead? Even if not, I'd rather have type-safety, friendlier function names and gaps in the ISA filled, but tastes differ.
In one place I was doing single-pass marching cubes on multiple FP32 scalar fields. A source data for every cube is 8 FP32 number in 1 AVX register. Sometimes I need to broadcast FP32 lanes from the vector, where the source index is known at compile time. For most of these indices, a combo of _mm256_movehdup_ps/_mm256_moveldup_ps + _mm256_permute4x64_pd is the fastest way to do so.
In another place I needed to broadcast FP64 values from non-zero lane.
In another place I wanted to avoid MXCSR error flag for 3D FP64 vectors:
// Duplicate Z into W to avoid division by zero
size = _mm256_permute4x64_pd( size, _MM_SHUFFLE( 2, 2, 1, 0 ) );
const __m256d b_xxy = _mm256_permute4x64_pd( b, _MM_SHUFFLE( 3, 1, 0, 0 ) );
const __m256d bw_yzz = _mm256_permute4x64_pd( bWeight, _MM_SHUFFLE( 3, 2, 2, 1 ) );
__m256d tmp = _mm256_mul_pd( b_xxy, bw_yzz );
Wasn’t related to complex numbers, I was doing something like that, saving a few instructions: https://en.wikipedia.org/wiki/Distance_from_a_point_to_a_lin...
> but #ifdef is increasingly infeasible now that SVE and Risc-V V are coming, no?
I think that’s wishful thinking like all these years of Linux on desktop, or rewriting everything in Rust. I think ARM is good enough for most applications, except very small niches (very low power, very price-sensitive, mostly embedded).
> Those can actually be bridged just fine
Yeah, but other things can’t. For one, ARM has 8-byte SIMD vectors. Quite useful for a library which implements non-trivial algorithms processing 2D vector data with tons of FP32 2D vectors. Another thing, this whole source file https://github.com/Const-me/DtsDecoder/blob/master/Utils/sto... implementing an equivalent of a single vst3q_s16 NEON instruction. That code is called in a loop which does not do much else and is inlined, i.e. these 9 shuffle constants stay in vector registers.
> I was doing something like that, saving a few instructions
Oh, nice one.
> I think ARM is good enough for most applications, except very small niches (very low power, very price-sensitive, mostly embedded).
Are you talking about the ARM arch vs Risc-V? That may be, but ARM is also getting SVE, and adding yet another #if for that is going to be expensive.
> Yeah, but other things can’t. For one, ARM has 8-byte SIMD vectors.
hm, Highway does support half vectors which map nicely to the non-q instructions. We also support 8-bit StoreInterleaved3 which is implemented similar to yours (more instructions but fewer constants): https://github.com/google/highway/blob/master/hwy/ops/x86_12...
In my experience, integer mul are the hardest to bridge, but at least for dot products even that is feasible.