Llamafile 0.7 Brings AVX-512 Support: 10x Faster Prompt Eval Times for AMD Zen 4 | Hacker News Reader

Llamafile 0.7 Brings AVX-512 Support: 10x Faster Prompt Eval Times for AMD Zen 4

phoronix.com

91 pointsmfiguiere17 comments

Well, I was looking for any excuse to upgrade to AM5, so I think this qualifies.

Question: if I already have a previous version .llamafile, is there a way to repack the model blob with the new llamafile executable? That is, say I have mixtral_llamafilev6 promote to mixtral_llamafilvev7?

From the release page

  - Prompt evaluation now goes much faster on CPU. For example, f16 weights on Raspberry Pi 5 are now 8x faster. These new optimizations mostly apply to F16, BF16, Q8_0, Q4_0, Q4_0, and F32 weights. Depending on the hardware and weights being used, we've observed llamafile-0.7 going anywhere between 30% to 500% faster than llama.cpp upstream.
  ...
  - Support for AVX512 has been introduced. Owners of CPUs like Zen4 can expect to see 10x faster prompt eval times.

Yes, they're just ZIP files that also happen to be actually portable executables, ehm, excuse me, αcτµαlly pδrταblε εxεcµταblες.

https://github.com/Mozilla-Ocho/llamafile?tab=readme-ov-file...

Sure enough. Pop open the archive and the gguf is right there. There is even a `.args` file where you could instruct it to launch a different model name.

I have tried out Llamafile and I think it is bloody great. The simplicity of it is commendable.

One issue I hope they overcome for Windows however, is being able to run the executable when it’s more than 4GB (a Windows limitation) as all the powerful models are WAY in excess of that. I believe they will figure out a nice workaround in time for Windows users.

I thought Windows could already run the llamafile executable separately and call whatever sized model blob?

I just had a look at their Github, and true enough:

“External weights are particularly useful for Windows users because they enable you to work around Windows' 4GB executable file size limit.

For Windows users, here's an example for the Mistral LLM:

curl -L -o llamafile.exe https://github.com/Mozilla-Ocho/llamafile/releases/download/...

curl -L -o mistral.gguf https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.1-GGU...

./llamafile.exe -m mistral.gguf -ngl 9999

I mean, devolving to 1 executable and 1 data file is less slick, but it doesn't seem so terrible IMHO.

Not clear how AVX-512 can provide 2x speedup on Zen 4, even more 10x (if they are comparing with AVX2 which is obvious assumption). Zen 4 does not really have proper AVX-512 units, and 10x means that there was no vectorization at all before?

Zen 4 has equivalent (in throughput) AVX-512 units with the best current Intel CPUs, for most AVX-512 instructions. It is a myth that Zen 4 is somehow slower. For most AVX-512 instructions, both Zen 4 and Emerald Rapids do two 512-bit instructions per cycle (Zen 4 and Zen 3 do four 256-bit instructions per cycle, while Intel does only three 256-bit instructions per cycle, so Intel shows a greater increase in speed when using 512-bit instructions only because it has a lower throughput for 256-bit instructions).

The only exceptions are the loads/stores from/to the L1 cache, which have double throughput on Intel and the floating-point fused-multiply-add units, where the most expensive Xeon SKUs can do 2 FMAs per cycle, while Zen 4 can do only 1 FMA + 1 FADD per cycle.

Zen 4 implements the BF16 instruction set, which is likely to increase the speed many times for any AI/ML workload that uses BF16. It also implements the VNNI instruction set, which will accelerate any inference that uses INT8.

Even when these dedicated instructions are not used, AVX-512 is usually much faster on Zen 4, by eliminating bottlenecks caused by instruction fetch and decoding and by using the better designed AVX-512 instructions.

Huh, seems you are right. I've read about "not really real" AVX-512 implementation in Zen 4, and also saw that on my workload turning on AVX-512 on Zen 4 indeed gives almost nothing (compared to x2 speedup on Intel). But now that you've written it, I understood that my only test was very specific, full of 32-bit FMAs, and I was comparing Zen 4 with Skylake-X which has 2 FMA units.

It is very likely that they compared newly supported BF16 instruction set with some other (float? fp16? quantized ints?).

If you make simple 1-to-1 transition from AVX256 to AVX512, the speedup is usually <2x, unless there is a major bottleneck in instruction fetch and decoding. Also AVX512 in Zen is still double-pumped. Regarding FMA, again, if you compare FMA in AVX256 and AVX512, latencies and throughput are the same[1].

Comparing performance between different datatypes is probably fine, but it should be stated directly. Unless "Owners of CPUs like Zen4 can expect to see 10x faster prompt eval times" means comparison with Skylake.

[1] https://uops.info/table.html?search=vfmadd132ps&cb_lat=on&cb...

Nice result, congrats Justine! The bf16 dot instruction replaces 6 instructions: https://github.com/google/highway/blob/master/hwy/ops/x86_12... A 3-4x speedup vs SKX sounds quite plausible :)

While the width of units is indeed 256, certain operations like shuffles have special treatment so it is more like hybrid design that focus on efficiency. In reality, it works quite good and better instruction density per bytes processed leads to sometimes even less energy consumption when a certain path is upgraded to AVX512 from AVX2.

And as the sibling comment mentions, AVX512 is not just about the width but also about newer instructions available at 128b and 256b widths as well under AVX512VL.

AMD has four 256-bit execution units, which are ganged into two 512-bit execution units.

Intel has only three 256-bit execution units, which are also ganged into two 512-bit execution units, by adding an extra 256-bit unit, which stays idle in 256-bit mode.

So the throughput for most register-register 512-bit operations is the same for Intel and AMD, except that AMD has a single FP64 multiplier vs. two FP64 multipliers on Intel and that the path to the L1 cache has double width on Intel (while AMD does one 512-bit load per cycle + one 512-bit store every other cycle, Intel can do two 512-bit loads from L1 + one store per cycle).

512 adds lots of new instructions

+1. There are useful instructions besides increasing width, which is in fact less interesting. That being said, I'd have to look into how this was measured.

I use my llamafile nearly every day.

Reply on news.ycombinator.com