AMD Upstreams AVX10_V2_AUX Support into LLVM Clang 24 | Hacker News Reader

AMD Upstreams AVX10_V2_AUX Support into LLVM Clang 24

phoronix.com

1 pointrbanffy5 comments

What heavy CPU workloads AVX10 V2 AUX could help accelerate significantly?

It has a smells of 'we need to push back our IP deadlines' thingy.

These instructions are part of the ACE set, so I assume they are useful for AI - there are format conversions, rounding, comparison, and, of course, operations on these formats in the vector and mask registers. Looking at the documentation, which is huge, gives me the impression this will improve quantized models the most, but this is a guess.

No other workloads? Really?

Because, if I am not too mistaken, inference hardware would do all that and directly on the silicon, namely much faster and using much less energy.

Or is this to run quantized models on consumer CPUs??? Weird, because, if I do run models on my consumer system, I will want the full weights of the frontier open weight models.

Intel has done a lot of work to make their CPUs better for running inference. I'm sure people will figure out other creative misuses for the instructions. One reason to run inference in-process on a CPU is latency - setting up a GPU or an NPU to run a job introduce latency and context switches. Running it in-process avoids that, which is nice for tasks where the AI part runs for a very short time in comparison with starting the job on a GPU over PCIe.

This is one of the reasons IBM introduced an AI accelerator and added specific instructions to use it on their Telum II processor - so that limited inference can run in-process in transaction processing with latencies measured in clock cycles.

I think these instructions are for the case where inference is a minor part of the time it takes to process some data on that server - in that case the server will be more effective by having more, faster memory than having to go across a PCIe but to a GPU that will be idling most of the time, or going over the network to a dedicated system.

Something does not add up there: inference on interesting models is already very slow by itself on consumer systems. "Sending to a GPU" is literaly "zero time".

Reply on news.ycombinator.com