Converting integers to decimal strings faster with AVX-512
lemire.me
lemire.me
In any case, I think this whole argument really misses another important thing, which is that the ISA and the vector width are separate here different. If Intel would just give us AVX-512 with 256b or 128b vectors, it would lead to a very big increase in use IMO, without having to have massive register files and vector data paths complicate things. The instruction set improvements are good enough to justify this IMO. (Alder Lake makes it a more complex case since they'd need to do a lot of work to unify Gracemont/Golden Cove before they could even think about that, but I still think it would be great.) They could even just half the width of the vector load/store units relative to the vector size like AMD did with Zen 2 (e.g. 256b vecs are split into 2x128b ops.) I'll still take it
[1] https://twitter.com/GPUsAreMagic/status/1256866465577394181/...
But that could be a bit of a chicken and egg situation, right?
https://old.reddit.com/r/emulation/comments/lzfpz5/what_are_...
https://travisdowns.github.io/blog/2020/08/19/icl-avx512-fre...
People have made some incredibly broad assertions about the technology as a whole on the basis of a first-gen product that lingered on the market for far too long because of the 10nm woes.
People also hang on Linus's words far, far too much. He's incredibly toxic and willing to opine about stuff he's been disconnected from for 20 years. Lisa Su is putting her money where her mouth is, AMD is making a billion-dollar bet on AVX-512 at a time when Intel has actually abandoned it in the consumer market entirely. Maybe she's doing it just for the server market, but that still means she thinks server-market customers are going to see an actual benefit on normal server tasks, such that it will be worth making every single customer pay more for a larger piece of silicon.
It's not the kind of thing you do "just to win a couple HPC benchmarks". Sorry, I trust the person with the financial stake in this discussion a lot more than I trust Linus's spit-takes.
[0]: https://en.wikipedia.org/wiki/AVX-512 [1]: https://en.wikipedia.org/wiki/Xeon_Phi#Knights_Landing [2]: https://en.wikipedia.org/wiki/Skylake_(microarchitecture)#Hi...
Initially AVX-512 was known as "Larrabee New Instructions".
This instruction set, which included essential features, which have been missing in both earlier and later Intel ISAs, e.g. mask registers and scatter-gather instructions, was developed a few years before 2009, by a team in which many people had been brought from outside Intel.
The "Larrabee New Instructions" have been disclosed publicly in 2009, then the first hardware implementation available outside Intel, "Knights Ferry" was released in May 2010. Due to poor performance against GPUs, it was available only in development systems.
A year later, in 2011, Sandy Bridge was launched, the first Intel product with AVX. Even if AVX had significant improvements over SSE, it was seriously crippled in comparison with the older AVX-512 a.k.a. "Larrabee New Instructions".
It would have been much better for the Intel customers if Sandy Bridge would have implemented a 256-bit version of AVX-512 instead of implementing AVX. However Intel has always attempted to implement as few improvements as possible in each CPU generation, in order to minimize their production costs and maximize their profits. This worked very well for them as long as they did not have serious competition.
The next implementation of AVX-512 (using the name "Intel Many Integrated Cores Instructions"), was in Knights Corner, the first Xeon Phi, launched in Q4 2012. This version made some changes in the encoding of the instructions and it also removed some instructions intended for GPU applications.
The next implementation of AVX-512, which changed again the encoding of the instructions to the one used today, and which changed its name to AVX-512, was in Knights Landing, which was launched in Q2 2016.
With the launch of Skylake Server, in Q3 2017, AVX-512 appeared for the first time in mainstream Intel CPUs, but after removing some sets of instructions previously available on Xeon Phi.
AVX-512 is a much more pleasant ISA than AVX, e.g. by using the mask registers it is much easier to program loops when the length and alignment of data is arbitrary. Unfortunately the support for it is unpredictable, so it is usually not worthwhile to optimize for it.
Hopefully the rumors that Zen 4 supports AVX-512 are true, so its launch might be the real start of widespread use of AVX-512.
From my point of view it's just an additional compilation pass of your SIMD kernels (written using github.com/highway; disclosure: I am the main author), plus shipping a few tens/hundreds KB extra code, plus ensuring your tests exercise that platform as well.
Xeon Phi was always a very targeted and limited product for HPC. Nobody was doing tons of performance-sensitive JSON parsing or emulating a PS3 or ARM on a Xeon Phi. Emulator people aren't going to target hardware that doesn't exist in their target market.
In practice the first time AVX-512 was accessible to the general public was Skylake-X, so 2017. Five years ago. And it was a server/HEDT product. And it had some weird performance regressions (like the downclocking or pausing when you use too many heavy instructions) that bumped it back out of a lot of codebases it might have hypothetically been useful.
The first consumer architecture that would have implemented it was massively late due to 10nm delays. So was the 10nm successor to Skylake-X that fixes the downclocking and other shortcomings.
Basically the answer here is "10nm". AVX-512 is a casualty of the eternal 10nm delays. Intel got stuck on Skylake forever, and couldn't push any of their other developments forward. They've had fixes for a while for a ton of the stuff people complain about, they just couldn't manufacture them at scale.
They are only just launching their first post-14nm server platform literally this year, and it's not even Intel 7/10nm ESF, it's Ice Lake based lol.
I have seen several accounts of them being inefficient (drawing much power, large chip area, dragging down the clock of the entire chip). They have also been criticised for making context switches longer.
There are some good ideas in there though. What I think Intel should do is to redesign it as a single modern instruction set for 128 and 256-bit vectors (or for scalable vector lengths such as where ARM and RISC-V are going). Then express older SSE-AVX2 instructions in the new instructions' microcode, with the goal of replacing SSE-AVX2 in the long term. Otherwise, if Intel doesn't have a instruction set that is modern and feature-complete, I think Intel would be falling behind.
Edit: I noted you said "redesign" which is an important point. I don't disagree in principle, I just point this out to be clear you can use the new ISA with smaller vectors already, to anyone who doesn't know this. No need to throw out baby with bathwater. Maybe their "redesign" could just be ratifying this as a separate ISA feature...
> drawing much power, large chip area, dragging down the clock of the entire chip
The chip area thing has mostly been overblown IMO, the vector unit itself is rather large but would be anyway (you can't get away from this if you want fat vectors), and the register file is shared anyway so it's "only" 2x bigger relative to a 256b one, but still pales in comparison to the L2 or L3. The power considerations are significantly better at least as of Ice Lake/Tiger Lake, having pretty minimal impact on clock speeds/power licensing, but it's worth nothing these have only 1 FMA unit in consumer SKUs and are traditionally low core count designs. Seeing AVX-512 performance/power profiles on Sapphire Rapids or Ice Lake Xeon (dual FMAs, for one) would be more interesting but again it wouldn't match desktop SKUs in any case.
It's also worth noting AVX2 has similar problems e.g. on Haswell machines where it first appeared, and would similarly cause downclocking due to thermal constraints; but at the time the blogosphere wasn't as advanced and efficient at regurgitating wives' tales with zero context or wider understanding, so you don't see it talked about as much.
In any case I suspect Intel is probably thinking about moving in the direction of variable-width vector sizes. But they probably think about a lot of things. It's unclear how this will all pan out, though I do admit AVX-512 overall counts as a bit of a "fumbled bag" for them, so to speak. I just want the instructions! No bigger vectors needed.
4x bigger; avx512 has twice as many registers.
The connection to the 32 micro architectural registers is quite loose.
https://twitter.com/InstLatX64/status/1114141314441011200/ph...
The shorthand is that there's basically "xeon phi" and "everything else". Each of those sets has strictly increasing capabilities with each product generation - Phi Gen2 is a strict superset of Phi Gen1, but Consumer Gen2 doesn't necessarily include all the instructions in Phi Gen1.
There's a few exceptions and nuances (in particular, this is generation, not chronological order), but that's really all you need to know in practice. If you want to refine that a little further, there's "consumer", "server" and "xeon phi" set and each of those is very well-ordered. Basically they just took the consumer line and added some neural network instructions to it for server.
But if you're a consumer, there is only one feature set you need to worry about, and they are strictly ordered within that series. Tiger Lake will be strictly better than Ice Lake, and so on. Well, until Intel just dropped AVX-512 entirely of course, but AMD is adding support in Zen4, so it's not going away or anything.
I get that people enjoy taking the piss, but there were literally 11 different "feature sets" in MMX/SSE that come up in a trivial search, and if you counted every single combination that any chip ever implemented as being a "level" there are most likely 30 or 40 "levels" in MMX/SSE. Some of those even got dropped and abandoned, others became basically mandatory for running modern software. It would be a big scary venn diagram assuming it could even be represented at all.
Today we don't even think about it (except for Phenom II owners, hi guys!), we just found the useful sets of instructions and everything implements them. When was the last time you worried about whether your processor had SSE4 POPCNT? Over time it's a non-issue.
AMD will implement their own subset of features in Zen4 that they think are most useful, maybe add a few of their own, and over time things will settle in exactly the same way that MMX/SSE did.
AVX512 is significantly more intensive than previous generation SIMD instruction sets.
It takes up a lot of die space and it uses a massive amount of power.
It can use so much power and is so complex that the clock speed of the CPU is reduced slightly when AVX512 instructions are running. This led to an exaggerated outrage from enthusiast/gaming users who didn't want their clock speed reduced temporarily just because some program somewhere on their machine executed some AVX512 instructions. To this day you can still find articles about how to disable AVX512 for this reason.
AVX512 is also incompatible with having a heterogeneous architecture with smaller efficiency cores. The AVX512 part of the die is massive and power-hungry, so it's not really an option for making efficiency cores.
I think Intel is making the right choice to move away from AVX512 for consumer use cases. Very few, if any, consumer-target applications would even benefit from AVX512 optimizations. Very few companies would actually want to do it, given that the optimizations wouldn't run on AMD CPUs or many Intel CPUs.
It's best left as a specific optimization technique for very specific use cases where you know you're controlling the server hardware.
Maybe it's time for AMD to do something about this, like when they took the initiative to create x86-64
512-bit registers, ALUs, data paths, etc, are all really really physically big.
The biggest problem, to my understanding, is that SIMD extensions are mostly targeted with optimized hand-written assembly code. It's fine to say that the SIMD is variable-width, but you still have to run this on a processor where the hardware is not variable-width. Not just the SIMD unit, but also the feedpaths/etc. And some things a "generic" framework might let you do, might have catastrophic performance consequences.
In practice a lot of this will be avoided by just knowing the hardware you're going to target and not doing the things that hurt performance, but that also completely misses the point of variable-width, you might as well just target the hardware directly at that point.
And actually your data may not be "variable-width" either.
Software is typically compiled for x86-64, but not necessarily for each specific x86-64 processor available. You often just get a generic precompiled binary. That's not really optimized for the machine you're running the software on.
Maybe you want to have some restrictions on the size of the vectors you're working with to make it more convenient for the hardware, like "should be a multiple of 4x64 bits", but I still think there could be big gains with variable width in most cases.
Then you need to write your loops in such a way that they work with a variable width: i.e., doing more iterations if the vector width is smaller.
AVX-512 is a better instruction set and many tasks can be done with fewer instructions than when using AVX or SSE, resulting in a lower energy consumption, even at the same data width.
The increase in size and power consumption is due almost entirely to the fact that AVX-512 has both twice the number of registers and double-width registers in comparison with AVX. Moreover the current implementations have a corresponding widening of the execution units and datapaths.
If SSE or AVX would have been widened for higher performance, they would have had the same increases in size and power, but they would have remained less efficient instruction sets.
Even in the worst AVX-512 implementation, in Skylake Server, doing any computation in AVX-512 mode reduces a lot the energy consumption.
The problem with AVX-512 in Skylake Server and derived CPUs, e.g. Cascade Lake, is that those Intel CPUs have worse methods of limiting the power consumption than the contemporaneous AMD Zen. Whatever method was used by Intel, it reacted too slow during consumption peaks. Because of that, the Intel CPUs had to reduce the clock frequency in advance whenever they feared that a too large power consumption could happen in the future, e.g. when they see a sequence of AVX-512 instructions and they fear that more will follow.
While this does not matter for programs that do long computations with AVX-512, when the clock frequency really needs to go down, it handicaps the programs that execute only a few AVX-512 instructions, but enough to trigger the decrease in clock frequency, which slows down the non-AVX-512 instructions that follow.
This was a serious problem for all Intel CPUs derived from Skylake, where you must take care to not use AVX-512 instructions unless you intend to use many of them.
However it was not really a problem of AVX-512 but of Intel's methods for power and die temperature control. Those can be improved and Intel did improve them in later CPUs.
AVX-512 is not the only one that caused such undesirable behaviors. Even in much older Intel CPUs, the same kind of problems appear when you are interested to have maximum single-thread performance, but some random background process starts on another previously idle core. Even if that background process consumes a negligible power, the CPU is afraid that it might start to consume a lot and it reduces drastically the maximum turbo frequency compared with the case when a single core was active, causing the program that interests you to slow down.
This is exactly the same kind of problem, and it is visible especially on Windows, which has a huge quantity of enabled system services that may start to execute unexpectedly, even when you believe that the computer should be idle. Nevertheless, the people got used to this behavior, especially because it was little that they could do about it, so it was much less discussed than the AVX-512 slowdown.
The "huge power hungry" execution units are in fact largely the same execution units used for AVX and AVX2 - the CPU simply uses 2 adjacent 256-bit AVX/2 units as a single AVX-512 unit: this is the so-called port 0+1 fusion. Only the larger shuffles and the (optional) extra FMA unit on port 5 take significant addition space.
In fact, the shared core die area for SKL (no AVX-512) and SKX (has AVX-512) is the same! There is some dead space on the SKL die due to lack of SKL support but it is small: you can bet Intel didn't ship 100s of millions of SKL and SKL-derivative chips with a big space-wasting AVX-512 unit which couldn't be used on it: they could have made a new, more targeted layout if that was the case.
Having gone through the process, I could understand how this would be an annoying feature that violated previous assumptions about how Intel CPUs work.
E-cores (Gracemont) don't support AVX-512, only P-cores (Golden Cove) support AVX-512
An efficiency core with AVX-512 is basically a Xeon Phi core (and that approach actually probably would be good at the compute-heavy workloads efficiency cores are optimized for), so that's not an absurdity, but they didn't do that. So at that point the answer is "neither", and they turned it off on their big cores. But apparently they left the unit there in the silicon eating up a fairly large amount of space (I think it's around 10-20% of the core).
The real question is why nobody thought of this before... did they have some hardware/software solution to handle the instruction set differences that didn't work out? The fact that it was shoddily disabled before launch, and left in programmers' guides and other documents, speaks that maybe there was something like that and somehow it didn't end up happening.
(You could trap instruction errors and mark the thread as big-core-only, the Linux kernel did that at one point for AVX instructions in general (to save registers/bandwidth/etc they wouldn't save the AVX registers unless a thread actually used AVX). But there's more problems that still have to be solved first - how does software know how many AVX-aware cores are available? How does software know which core their CPUID instruction will be executed on? etc etc.)
If they had a heterogenous-ISA mechanism or something that will be introduced later, why isn't it on the roadmaps for next-gen/etc. And they didn't have those mechanisms, then why is it still on the core on an extremely high-volume consumer architecture?
You could say maybe it's a testbed for Sapphire Rapids, but every company has dedicated test silicon for that, and 10-20% of every single consumer chip sold is a huge waste of money. Sure, they'd have to design a core without it, but Intel absolutely has the money to do that - just like Skylake vs Skylake-X. The unit is designed to be modular so it's not hard, and they'd make it back extremely quickly.
Maybe they just needed some dead silicon to spread out the thermal load a bit? Obviously dead silicon has to be dead, so it doesn't matter if you print AVX-512 or the names of the design team, it's not supposed to be powered up either way.
Anyway that's my take - the "why take it back off" part is straightforward, the answer is big.LITTLE. The real interesting question is "so why is it still on the silicon".
Linus has mentioned he want AVX-512 to die a gruesome death. AMD also doesn’t actually support AVX-512 (but when seen run it in 2 cycles of 256-bit wide instruction aka AVX-2.)
The whole AVX-512 situation is just so complicated in hardwares and hence softwares (which subset to enable?) and this is probably the primary reasons for the lack of adoption, even from Intel themselves.
Eg the disabled AVX-512 that you alluded to in recent Intel CPU is because of its hybrid nature and the little CPU aren’t capable of AVX-512 and since they want to same instructions to be able to run on both big and little CPU, the only move is to disable it. Anandtech tested with the little CPU disabled in BIOS, and while Intel said the AVX-512 is fused off, they actually are able to run AVX-512.
github.com/highway compiles the same source code to AVX2 or AVX512, and checks at runtime which to use. I've seen 1.5x speedups from this (even including throttling on 5 year old Skylakes). Why not use what's there?
Caveat 2: I am a user of Intel Xeon Phi (KNL specifically) on an HPC system. So my view is biased in this way. It's probably right that the extensions has much less variants from non-Xeon Phi line of products. But, it still carries some weights, because it is a sign that Intel was struggling to strike a balance at least in the beginning. They wasn't sure what to offer, the fact that Xeon Phi line is killed and replaced by GPU is their admission of failure there.
Then vendor other than Intel (AMD) would hesitate to adopt that because even Intel wasn't sure what to offer. And if AMD doesn't pick that up, the usefulness of AVX-512 is in doubt. E.g. in the traditional way of prepping binary releases (many distribution methods still do) is to target the "highest common factor" of supported instructions of reasonably recent CPUs, meaning they would not target AVX-512 at all as the latest CPU from AMD doesn't have it (they later add fake AVX-512 capability.) Now it means that whether AVX-512 is useful to you as an end users then depends on the distributions a lot (and in practice none of the general purpose binaries are targeting this. In the rare case they build using the Intel compilers, it is very easy to ship both, which is not true if using GNU compilers.) And if it ain't going to useful for the majority of users, there's no hurry to upgrade specially for AVX-512.
Then it becomes a disadvantage for Intel to increase the cost of making AVX-512 capable CPUs to compute in the lower ends. Note that there are other comments and articles on this already—AVX-512 is not cheap. It costs significant die area which is cost.
That further complicates the things when Intel is introducing BIG.little design, where in the little CPU, low energy draw and low cost hence smaller die size drives towards having no AVX-512. And since the only way to guarantee any binaries can jump between the BIG and little CPU, they choose to disable AVX-512 on the BIG CPU even if its there. (AnandTech article is much more detailed about this.)
P.S. I'm sure historians would give a better and more balanced accounts on why it fails, but I think it is safe to say the standard is now too fragmented, not necessarily in implementation sans Xeon Phi but in the market, it is probably remains to be a niche.
P.S.2 A fun fact is that in the recent years AMD's CPU are praised by their performance to watt ratio because of the advances in fabrication tech (basically from TSMC's advances). But I did some calculations earlier that if we're only talking about FLOPS, and compares FLOPS per watt with the consideration of AVX-512 units (many Intel CPUs even have 2 AVX-512 units), it is actually comparable if not lower for Intel. (That was an old calculations and with the newest Intel getting on better fab. I'm sure the story has changed already.)
But then no much people if any (I heard none, but I don't read too much except sometimes from AnandTech) has mentioned this and make the comparison "fair" to Intel. Why? Because the end users most of the time aren't going to be benefited by AVX-512 (again because of distributions) so that's often not taken into accounts, sometimes indirectly by performing benchmarks that is not AVX-512 aware.
Fair point about heterogeneous cores. I suspect those aren't going away. Perhaps the industry should move to a model where threads are temporarily pinned to their core while running SIMD kernels, and re-checking the current CPU capabilities rather than storing the result. That would allow using both AVX2 and AVX-512 depending on which core is active, and doesn't require OS changes (except to OSX which AFAIK still cannot reliably pin threads), right?
I totally agree that AVX-512 is helpful for power efficiency. Without it, I struggle to see how Intel/AMD are going to compete with current/future ARM and RISC-V.
Those constants being used look a lot like a table.
And permb_const is a list of indices for the result characters within the vector register - the algorithm works on 64-bit integers, but the result must be a list of bytes, so it picks out every 8th one.
There are constants, but they are used directly as arguments for the AVX-512 intrinsics. They must still be loaded into the AVX-512 registers from memory, but they are loaded from locations already known at compile-time and the loads can be scheduled optimally by the compiler, because they are not dependent on any other instructions.
For a table stored in an array in memory, the loads can be done only after the indices are computed and the loads may need to be done in a random order, not in the sequential order of the memory locations. When the latencies of the loads cannot be hidden, they can slow a lot any algorithm.
This will never show up in a microbenchmark where the function's instructions are always hot. In fact, a lot of microbenchmarking software "warms up" the code by calling it a bunch of times before starting to time it, to maximize the chances of them being able to ignore this reality.
In contrast, for the scalar code, the CPU must complete the divisions (which'll become multiplications, but even those still have relatively high latency) before it can even begin to look for what to cache.
To remove leading zeroes, you'd need to use one of bsf/lzcnt/pcmpistri, and a masked store, which has some cost, but still stays branchless and will probably easily be compensated by the smaller cache/storage/network usage & shorter decoding.
He could at least have made a proper assembly version for a proper comparison.