AVX10/128 is a silly idea | Hacker News Reader

AVX10/128 is a silly idea

chipsandcheese.com

136 pointspicture93 comments

So much of this, such as the backpedaling on support of AVX512, seems to be at least indirectly related to the desire to "solve" the problem that occurs when Intel wants to have little (efficiency) cores that might not have the same feature set as the big (performance) cores.

Two decades ago I put a Pentium III and a Celeron on to the same dual processor motherboard (using a Celery to Nintendo cartridge adapter), and said, "Look! Different processors can run at the same time in the same system. Perhaps we should have a way for processes to express a desire to use specific CPU features, and if they request features not available on all CPUs, we'll keep them on the ones that do." Sigh.

Intel should've told OS writers years ago that heterogenous CPU features would be a thing soon, so prepare for it. Then we could've easily had AVX512 on big cores and AVX256 on little cores, and it'd've all be just fine.

No it wouldn't because promotion of a process from an E core to a P core isn't free. Microsoft and Intel could literally do this today on existing hardware by trapping unsupported instructions and then having the OS scheduler promote the process but they don't because it takes so stupidly long and requires an immense burst of resources. All this AVX10/128 is a necessary evil because the far better option is for an E core to be able to at least run the code, even if it's not performant.

Or, intel could do like AMD and just design P cores that are space efficient enough to stuff them into a chip like E cores while still getting good performance.

How long is "stupidly long"? I get that it's expensive to transition back and forth repeatedly, but if it's on the order of even tens of milliseconds, you only need to do it once and then forever pin that process to P cores. And you could easily have a preamble in the init that checks the instructions you need to trigger that on boot if you wanted to front-load that cost. Heck, you could have it be a filesystem attribute that's automatically set when the promotion is noticed so that you just do it with 0 overhead after the very first execution.

Executable formats like ELF and PE already allow pretty much arbitrary tags - just define one that says "This executable uses instruction set extension X" and you don't even need the cost of a trap or modifying the file/filesystem metadata at runtime.

Sure, old apps might not then have that flag, but old apps aren't using avx512 either.

I’m sure there are Avx512 apps that wouldn’t have this flag which poses a problem for Linux in terms of back compat.

this is already done by libraries and how new instructions are leveraged on cores that have them. During init a library checks availability of instructions and sets function pointers to relevant routines.

Yeah, and you could set the process cpu masks at init time in your own code and do the equivalent. Hell, Intel could add that init step as a flag for their compiler.

As mentioned elsewhere, I'm not sure this is hardware design being forced to adapt due to software implementation difficulties, so much as avx512 as a whole may not be worth the hardware area going forward. Even on the larger cores.

Great then all processes will be running on the P-cores only, with the E-cores idling all day.

Note that even simple things like memcpy, which every single program out there will use, will use AVX512!

Generally it seems things that benefit from super wide simd also benefit from threading. If productivity apps want to leave performance on the table, so be it.

But really I think the entire ecore/pcore split for avx512 is academic, as I'm not sure tying features and flexibility to wider registers makes sense even on pcores, as I'm not sure the hardware area cost is worth the benefit. I honestly wouldn't be surprised if newer architectures don't have the wider registers options on even their larger cores.

You're chasing a pretty small market IMHO of people who have datasets large enough to benefit from large registers and wider alus, but not so large it's worth it to pass it over to an even more specialized accelerator.

The benchmarks that tend to show 512-wide simd benefits may get even bigger benefits from running on a GPU.

Actually not, on "recent"/"not too old" x86_64 architecture, "rep movsX" instructions are now hardware accelerated for short and big memory blocks. Memcpy and other basic memory operations are getting their code generated directly by the compiler which shortcuts the OS runtime.

> Microsoft and Intel could literally do this today on existing hardware by trapping unsupported instructions and then having the OS scheduler promote the process but they don't because it takes so stupidly long and requires an immense burst of resources.

Why not just determine the right cpu to run on by examining the arch of the binary? Waiting for an instruction failure seems ridiculous.

Because you don't want any AVX512 binary to get stuck on a P-core forever.

Also, if application developers got to choose all of them would build their apps to request the most performance. And then how do we put anything on an efficiency core if every application claims to need to run on a P core. Back to square one it is.

Why not use a prominent OS display "power efficiency" so the user can harass devs about a calendar app running on P core

Because your calendar app happens to use a jpeg decoding library that uses "high power" instructions.

Because the binary might contain a JIT, or even more simply, it might load a DLL with new code.

Both the library and the AVX512 generating JIT binary would have the arch set to AVX512 in the executable format header.

Well, but what if it doesn't? Also, who's supposed to set that bit? The programmer, or the compiler? In either case it's possible to set it wrong.

If the compiler generates the instructions, the compiler has to set the correct arch, if the JIT does that, the author has to override the arch in the build scripts.

If the arch is incorrect, it is a bug, and program will crash on illegal instruction. Ie like if you had an ARM binary that was incorrectly set to x64 and was run on x64.

This isn't an architecture, though. It's an instruction set.

What if the JIT generates new instructions dynamically and initially there are no AVX512 instructions but later on in the process' lifetime there are?

Then it crashes with an illegal instruction exception.

The instruction set is the major part of an architecture spec from the point of view of an userspace application. You can design the binary header flags to express instruction sub/set when you find that reasonable. In any case, regardless if the instructions are hardcoded to the text or generated during the runtime, you need to communicate the intention to use the specific instruction set to the kernel, so it can schedule your process on appropriate cores, and check if there are such cores present in the first place.

Microsoft’s implementation of C standard library routines like `memcpy` is checking for a global flag, and use AVX when available. The flag is initialized on startup by testing the output of `cpuid` instruction.

For this reason, most binaries built with modern Visual C++ are technically using AVX instructions even when compiled for SSE only. It doesn’t mean these binaries gonna fail without AVX support, it only means they’re capable of using AVX when available.

AVX 10/128 doesn't sound particularly useful, because you still need to do AVX2 at 256, so you're at best skipping half the 256-bit registers. But still doing some 256-bit calculations (even if it's double pumped)

I'd think you could do some pretty good heuristics, like if the thread hit P-core instructions in the last time slice, don't schedule it on an E core for the next one. When a time slice starts on a P-core, leave the specialty instructions disabled, so you can monitor usage --- if you trap, enable it, and return; if that cost is still too high (which it might be), keep track of how many time slices in a row hit the trap, and maybe enable the P-core instruction preemptively for a few slices.

Or just, give the program more information and let it decide. If you've got some cores with avx and some without, maybe the process wants to schedule only on the avx cores. Or maybe it can schedule some threads on any core and others need an avx core. As long as the possible permutations at run time aren't too crazy, it's reasonable.

The problem with that is that you'd then have processes that could only run if a big core was powered up. The main point of having little cores is so you can power down the big ones, so you want to be able to run anything on a little core that can run on a big core.

FWIW, I remember Intel manuals years ago warning that CPUID results might change across cores, but it seems software did not prepare :)

Great writeup! It sounds like the author had quite the ordeal with Intel Knights Landing; often, the most informative articles come from genuine experiences (and sometimes, pain).

I'm in the trenches with AMX right now, and it's not without its challenges. When Nvidia rolled out the `wmma` intrinsics for tensor cores years ago, they felt a bit unconventional. But as technology evolves, these new extensions seem to be increasingly idiosyncratic, challenging to navigate, and hyper-specialized.

Hardware improvements significantly outpace compiler optimizations. So hand-written kernels are becoming the norm again :)

Can you please elaborate the last paragraph? Sounds interesting that we can now legitimately write raw assemblies! I always want to do that but it's still in my backlog.

I think they mean writing code using compiler intrinsics, which are one layer above assembly (and how you should write most “assembly” these days). Many of the modern intrinsics in silicon are sufficiently complex in how they manipulate the data that it is effectively impossible for a compiler to detect applications of those instructions to normal idiomatic code. If you want to use those instructions, you will have to use them directly because the compiler can’t do it for you. The nice thing about intrinsics is that they look like C functions so they are easy to use.

There is definitely a growing cottage industry for writing high-performance kernels using this increasingly rich set of intrinsics. I find myself writing a lot more intrinsics code than I used to.

Writing assembly is rarely a good choice in modern days. Whether assembly code is correct or not depends not just on the target CPU, but also on the target OS.

See that bug for the example what gonna happen https://bugs.chromium.org/p/chromium/issues/detail?id=121838...

TLDR: Windows ABI conventions say xmm6-xmm15 vector registers are non-volatile. When a function uses them, it needs to backup old data, and restore before the return. Linux ABI conventions say all vector registers are volatile and don’t need to be preserved. Chromium has non-trivial pieces written in assembly, and crashes.

When writing C or C++ with intrinsics, registers are managed by the compiler. Modern ones follow these ABI conventions very carefully.

This is why having JIT/AOT toolchains is great, let the runtime optmize to what is actually available, instead of shipping multiple dlls/so with exponential complexity of ISA mappings.

Every Android phone and ChromeOS device is its own special snowflake in hardware configuration, regarding RISC, ARM and Intel CPUs, or GPUs for that matter.

Speculating a bit here: Perhaps one could even create a JIT that generates different code for Intel P-cores and E-cores in the same process. However, that would be very challenging to get working if the OS can switch the process from one core to another at any time.

The runtime would basically have to detect which AVX-512 instruction has failed at any point in the compiled code, map the processor state to AVX-2 registers and jump to an equivalent AVX2 code path. It would probably be easier to build the implementation on software transactions that can be restarted at coarse granularity, i.e. something like STM.

How do you imagine Android works across high and lower power ARM cores?

However ART is also a great piece of engineering at JVM and CLR level, even though Dalvik was quite lame, even when compared against Nokia and Sony-Ericson J2ME implementations.

ART is another level.

Nowadays you have an handwritten Assembly interpreter for faster startup, a JIT compiler, that also gathers PGO data, and an AOT compiler that picks up that data when the device is idle taking all the required optimizations.

To take this even furter, those PGO files are uploaded into the Play Store, and shared across compatible devices when applications are downloaded, not only allows jumping straight into JIT with PGO data, all devices collaborate gathering PGO data that covers all common use cases across the application audience.

And yes this also takes into account the ARM architectures.

On the JVM side, IBM and Azul have similar offerings with cloud based JITs for their implementations.

I thought ARM big.little implementations supported the exact same instruction set on both big and little cores. Don't they?

No, they are different ARM ISAs.

https://www.arm.com/technologies/big-little

big.LITTLE allows a hybrid microarchitecture, but all cores must support the same instruction set architecture.

Even SVE must use the same vector width across all cores, which makes it unlikely that >128b SIMD will ever show up on consumer cores in the near future: https://gist.github.com/zingaburga/805669eb891c820bd220418ee...

No they don't, picking the first example "Armv9.2 - DynamIQ big.LITTLE - DynamIQ Shared Units-120"

https://developer.arm.com/documentation/102484/0001/The-Cort...

https://developer.arm.com/Processors/Cortex-A720

https://developer.arm.com/Processors/Cortex-A520

You will notice that for e.g. Asymmetric MTE isn't supported across all of them, and the differences in microarchitecture are enough to have impact in the kind of instructions being used.

Your first link comes up as blank for me (documentation-service.arm.com returns 404 for me).

But it's worth pointing out that X4, A720 and A520 support asymmetric MTE [1] - just the A720 page you linked to fails to mention it.

[1] https://www.arm.com/blogs/blueprint/memory-safety-arm-memory...

I'm not sure the parenthetical "including Asymmetric MTE" in one doesn't mean it isn't present in the other, especially since the lists are otherwise in different orders and just generally seem to be written by different persons.

The idea of big.LITTLE is that processes can be transparently moved between different cores by the OS, so any differences in ISA would need to be virtualised away by the operating system - or else you get crazy bugs like https://www.mono-project.com/news/2016/09/12/arm64-icache/

Anyone that has done microarchitecture debugging with VTune knows those details matter.

I attended this db seminar at Berkeley earlier today… Helena Caminal (Google) – “Vector architectures for data-parallel workloads”

https://sky.cs.berkeley.edu/events/database-seminar-helena-c...

Very interesting work about associative processing: “The Associative Processing (AP) paradigm leverages content-addressable memories to realize bit-serial arithmetic and logic operations, via sequences of search and update memory operations over multiple memory rows in parallel.”

Interesting, I have seen claim by Adobe senior principal scientist, from 2012[1] and then 10 years later in 2022[2] that if you are writing single threaded scaler code in vanilla C++ you can only get up to 0.25% of maximum GFLOPS of typical machine. Looks like in 10 years nothing really changed. In my understanding this is real problem that needs to be immediately solved, not just for C++ but any other language.

[1] Sean Parent: "Now What? A vignette in three parts" https://www.youtube.com/watch?v=iGenpw2NeKQ&t=1382s

[2] Keynote: The Tragedy of C++, Acts One & Two - Sean Parent - CppNorth 2022 https://youtu.be/kZCPURMH744?t=2924

This article highly underestimates the value of keeping 128b vector performance high. Most code doesn't get recompiled or compiled with the appropriate flags. There is significant overhead involved in supporting 1x512b operations, 2x256b operations, or 4x128b operations per cycle with the same datapaths, forwarding network, and register files. Until 128b vector performance gets deprecated this tension incentives narrow implementations.

> There is significant overhead involved

Current CPUs already do this, and have been doing so for quite some time. And AVX10/128 doesn't alleviate it either.

Unless Intel is planning on deprecating SSE then this is already baked in and AVX10/128 existing or not existing won't be changing anything.

For string parsing and formatting or codec workloads (or for swiss tables, but really these are slower than alternatives so nobody should use them) it may be better to target 32x 128b registers than to target 16x 32b registers. An implementation that aliases the two or has high runtime costs for switching between the full set of 128b registers and the reduced set of 32b registers would be exceedingly annoying though.

64bytes is a sweet spot, a cache line on x86_64, going above seems "not appropriate".

I'm always fascinated when engineers who make my year's salary in a week come up with something as dumb as this.

Intel: "We've unified the many different vector instruction sets into a single standard!"

Me: "Page one of the unified standard lists its many variants, which seem to be the same number that existed before?"

Intel: "We've renamed them all! See! Unified!"

And vector is a misnomer.

ARM SVE2 and RISC-V Vector 1.0 are vector extensions. AVX is just SIMD with marketing.

It's not?

I'm confused as to what the difference is? I thought SIMD = vector.

Traditionally "vectors" are variable size while SIMD is fixed size. Almost no one uses those definitions any more.

To add on: SVE/RVV also have fixed size vectors. It's just that the size is specified by the hardware, whilst for AVX, software specifies the size.

This is at least somewhat wrong at best - SVE2 doesn't have a dynamic vector length like RVV (only non-fixed hardware register width, and masking), and Cray-1, said to be the first vector processor, has a fixed hardware register size (64 elements, each 64 bits).

So either SVE2 isn't vector due to lack of VL, or AVX-512/AVX10 is vector if VL isn't necessary (the only difference between AVX and SVE2 being the scalable registers, which Cray-1 doesn't have).

Furthermore, IMO calling an ISA only one of 'vector' or 'SIMD' based on whether it has VL doesn't make sense - semantically, VL is a subset of masking (if you want VL & a mask, just and them), whether hardware will short-circuit on it entirely depends on the hardware, and could be done on both. All RVV hardware that I know of takes the same amount of time for VL=1 and VL=max. (and defining vector vs SIMD by existence of non-fixed register width excludes Cray-1, which you do not want to do)

> SVE2 doesn't have a dynamic vector length like RVV

SVE and RVV are no different in that regard. Note that RVV's VL doesn't alter the vector length - it's just an alternative way of masking (which can easily be implemented in SVE).

Yes, I agree that VL isn't meaningfully distinct from masking as I noted in a sibling reply; regardless, some definitions of SIMD vs vector rely on it, e.g. wikipedia: https://en.wikipedia.org/wiki/Vector_processor#Comparison_wi...

This doesn't sound quite right - RVV has 100% fixed length vectors that can't vary in length.

If setting VL is enough to qualify (which seems to be such a minor differentiating factor), then I'd argue that SVE also qualifies since you can use WHILELT to do exactly the same thing as setting VL. SVE2.1 basically makes the whole thing identical, with its support for predicate-as-counter.

Again, I pretty much agree. WHILELT makes RVV-equivalent things trivial, but it's just a data→data instruction that could be done in AVX-512 too, as it's just a scalar (1<<n)-1, so the SVE2 & AVX separation in snvzz's original message is still unexplained. (assuming they don't mean to exclude Cray-1; unless they did mean SVE2.1? or maybe actually AVX (AVX1) and not AVX-512/AVX10? though even there you can still mask things to some limited extent)

SVE2.1 predicate-as-counter seems interesting, but I can't find any overview of it (only specific instructions/intrinsics), would you happen to know of any?

> would you happen to know of any?

I doubt I know any more than you. ARM's info on the matter is a little scarce, and I struggle to understand their documentation at times.

3 hierarchies of featuresets in which each implies all prior features are available seems better than the current situation: https://fuse.wikichip.org/wp-content/uploads/2019/12/avx512_...

I agree, but so is AVX10/256, get rid of that also thanks:)

why? As I understand it 256 bit avx has been standard on x86 CPUs for generations. avx10 in this context just changes the number of instructions that can be done with 256bit registers. seems like it would be pretty silly to remove 256bit avx that is already useful.

> why? As I understand it 256 bit avx has been standard on x86 CPUs for generations.

The Atom-derived cores did not support AVX for quite a long time. Even the Tremont microarchitecture (i.e. Elkhart Lake launched in September 2020 and Jasper Lake launched January 2021) does not support AVX. Only from the Gracemont microarchitecture on (launched in November 2021), there exist Intel Atom processors that support AVX/AVX2.

AVX was also unavailable on Pentium and Celeron branded processors but I think that has changed.

These are based on the same microarchitecture as the Atom cores, so it was also a problem up to Tremont.

With the Gracemont microarchitecture, this "principally" changed, but until now Intel has not announced any processor of this microarchitecture that it branded "Pentium" or "Celeron":

> https://en.wikipedia.org/w/index.php?title=Gracemont_(microa...

Are they? (Were they?) I assumed they were the main Core architecture just with features removed, either by design or after production.

Both are true.

Some Atom line processors were branded Pentium/Celeron. The low end big core SKUs were also branded Pentium/Celeron (and often had features disabled).

You are misunderstanding, avx10/256 adds nothing, all those instructions already exist under avx512. All avx10/256 does is create yet another target, would be better to unify under avx512(which already supports 256 & 128 registers as a requirement).

I don't agree. A 256 bit profile of the new AVX stuff gives you lots of goodies over AVX2, including mask predication, the fancy ternary op, and so on.

Additionally, because it consumes half as many register bits as the 512 bit profile, it could be a lot more economical (and power efficient) to implement on cores that are not the high end server parts. Keep in mind ARM (even the top of their line) is still stuck on 128.

So I think the case for this is a good one.

To clarify for readers, since I'm sure raphlinus understands this, those 256-bit instructions are present in AVX-512. But it will be nice to have those 256-bit instructions (my favorites are vpcompressb and friends) on cpus that do not have 512-bit instructions.

AVX10/256 allows you to use the EVEX encoding with 256-bit registers without assuming that you'll have 512-bit register support. And if I have to be honest, it's all the things that EVEX encoding adds (double the number of registers, per-instruction rounding modes, masked vector operations) that are interesting, not the fact that you have 512-bit registers.

Those things are nice but in the end if I'm trying to get the most work done in the shortest amount of time, having wider registers help more than that stuff.

masked operations are really useful for dealing with the tail of loops. You can add a single masked instruction to deal with the last few elements rather than having to do a scalar loop at the end to deal with the bits that don't fit nicely into a register. Doubling the number of registers is also great. It's basically free (since CPUS have lots of virtual registers) and can lead to way fewer load and store instructions. Most importantly, these changes are basically free from an area and power perspective. A 2x wider register will give you ~1.8x better performance, but it typically comes with ~1.5x higher power and wasted silicon for non-vectorized code. EVEX gives you all the really nice to have features that might only give you 20% better performance, but do so while lowering power and not taking up extra die area.

Not necessarily. The last I checked, even Intel's compilers default to 256-bit vectorization when compiling against AVX-512 targets. And of course, AMD implemented AVX-512 via double-pumping of 256-bit SIMD operations (so AVX-512 isn't really faster than AVX-256 there).

If memory bandwidth is a significant factor in your execution time, doubling the SIMD vector size is going to give you far less than the 100% theoretical speed boost (before taking into account thermal throttling, of course). Furthermore, the execution penalties due to things like remainder loops or vector misalignment (every misaligned 512-bit vector crosses cache lines) are comparatively more painful. The fancy new features ameliorate those downsides a lot and furthermore make vectorization profitable on a larger fraction of your code.

And finally, if the limiting factor of your code's execution time is the number of FMA units you can ram data through, then at this point, you've already likely switched to using a GPU to do that instead of the CPU.

Hopefully RISC-V will take off in high performance, before any of this is relevant.

Even if it does, this is also highly relevant to desktop CPUs.

>highly relevant

It is a SIMD extension, despite Vector era (RISC-V Vector 1.0, ARM SVE2). Far from relevant.

>desktop CPUs

Will too be RISC-V. Not far from now.

Some chips (likely from Intel and AMD) will offer "x86 acceleration" to run legacy software, for a transitional period lasting a few years.

Before making bold claims, it might be useful to get more familiar with a mess that are myriad of RISC-V extensions and seemingly insane/backwards decisions it makes completely ignoring the hard lessons learned by both ARM and x86.

>it might be useful to get more familiar with a mess that are myriad of RISC-V extensions and seemingly insane/backwards decisions it makes completely ignoring the hard lessons learned by both ARM and x86.

FUD unless you give specific examples and evidence backing them.

Many "seemingly insane/backwards decisions" in RISC-V are good decisions backed by large bodies of data.

Refer to Computer Architecture: A Quantitative Approach[0].

0. ISBN-13 978-0128119051

Previously: https://news.ycombinator.com/item?id=32053592

Somebody disagreeing with RISC-V design, news at 11h.

Just wait for RISC-VI, it'll be great

Judging by the PC's history of transitioning from 32-bit to 64-bit x86 (still not complete after 20 years), and from x86 to ARM (Windows on ARM is at 14% market share after more than 10 years)… well. I'm just a bit skeptical.

Though to be fair, it all depends on the hardware. Windows on ARM struggled not just because it wasn't x86 but because nobody was making powerful-enough ARM processors. Now that those processors are starting to arrive, it's supposedly going to take off in the next few years. …Probably only on the low end, though. But you never know.

And RISC-V? Maybe someday.

>(Windows on ARM is at 14% market share after more than 10 years)

Is it really that high? Where do these figures come from?

According to that research https://www.counterpointresearch.com/insights/arm-based-pcs-... 14% of laptops currently being sold have an ARM CPUs.

However, 90% of them are Macbooks, which means the share of the ARM laptops which run Windows, Linux or ChromeOS is 1.4%.

I don't think Win ARM market share is at 14%. Where did you get that stat?

For SIMD kernels that do "general-purpose computation" like formatting strings, parsing documents, or entropy encoding and decoding, it's difficult to use variable-length vector extensions, and ultimately these sorts of things written for these targets are likely to branch at runtime based on the actual width reported by the CPU. One of the most important operations for this sort of thing, TBLQ, doesn't even exist in SVE prior to SVE2.1 published ~1 year ago, so parser devs targeting aarch64 can safely target NEON until SVE2.1 lands in the hardware they care about.

riscv is just an instruction set. in a lot of ways its not as nice as AArch64 or x64. I don't see a strong impediment to having a world-class riscv processor, but it hasn't happened yet and the only reason to pick riscv vs arm is that you're a bottom-feeder, don't want to pay arm licensing fees. People who will switch architectures to save a few cents aren't people who make or buy world-class cpus.

I think you've just convinced me that RISC V is the way to go, long term.

Historically, bottom feeders won in IT. unix was a bottom feeder againts the IBM os'es and VMS. x86 was a bottom feeder architecture against mainframes, server class hardware, and workstations. Linux was a bottom feeder against the real UNIXes like SunOS and HPUX.

Bottom feeders get picked as cheapest option, then client demand of better features slowly pushes them upwards. Meanwhile the top concentrates on its most profitable customers, showing disdain for bottom feeders until they've grown out of control.

The last 5 Tandem owners are probably still out there somewhere, showing all of us what real uptimes look like.

Reply on news.ycombinator.com