Hacker News Reader

Comment by kpw94 | Hacker News Reader

This is incredibly easy to setup, just tried it for first time.

How fast is it supposed to go?

Just tried on linux with `cmake .. -DGGML_OPENBLAS=ON` on a AMD Ryzen 7 5700g (no discrete GPU, only integrated graphics)

    ./bin/sd -m ../models/sd-v1-4-ggml-model-f32.bin -p "a lovely cat"
    [INFO]  stable-diffusion.cpp:2525 - loading model from '../models/sd-v1-4-ggml-model-f32.bin'
    ...
    [INFO]  stable-diffusion.cpp:3375 - start sampling
    [INFO]  stable-diffusion.cpp:3067 - step 1 sampling completed, taking 12.25s
    [INFO]  stable-diffusion.cpp:3067 - step 2 sampling completed, taking 12.22s
    [INFO]  stable-diffusion.cpp:3067 - step 3 sampling completed, taking 12.56s
    ...
    sampling completed, taking 246.40s

Is that expected performance?

(EDIT: Don't have open Blas installed, so that flag is no-op)

View on HN

This is nice, it basically does what i asked a year ago[0] and at the time pretty much every solution wanted a litany of Python dependencies that i ended up failing to install because it took ages... and then i ran out of disk space.

No, really, this replaces literal gigabytes of disk space with just a 799KB binary. And as a bonus using the Q8_0 format (the one that seems to be the fastest) it also saves ~2.3GB of data too.

That said, it seems to be buggy with anything other than the default 512x512 image size. Some sizes (e.g. 544x544) tend to cause assert fails, sizes smaller than 512x512 (which i tried as 512x512 is quite slow on my PC) sometimes generate garbage (anything smaller than 384x384 seems to always do that).

[0] https://news.ycombinator.com/item?id=32555608

Also got segfault core dump with different sizes. 512w * 768h worked

> sizes smaller than 512x512 (which i tried as 512x512 is quite slow on my PC) sometimes generate garbage (anything smaller than 384x384 seems to always do that).

Not sure about the speed, but the garbage output might be due to the model instead of the library. I’ve always got garbage (using other tools) when I tried 256x256.

SD 1.x does not work well at resolutions less than 512x512, it's only really trained to output 512x512.

IME, SD 1.5 base model, IIRC (I don’t use the base model much) works tolerably well with the larger dimension at 512 and the smaller dimension between 384-512, and quite a lot of SD 1.5 based checkpoints work pretty well up to a larger dimension of 768 (some with smaller dimension up to that, too.)

But SD also has hard requirement that the output dimensions be divisible by 8 (I believe as a consequence of the pixel space to latent space ratio being 8:1 in each dimension) so its not surprising that 544x544 fails hard.

You should quantize the model, but 12s/iter seems about right.

Nice. Tried the fp32, q8_0, and q4_0, and for some reason they all take ~12s/iter.

Must have something wrong with my setup, but no big deal, for my minimal usage of it, and the amount of time spent, fp32@12s/iter is fine

Hmm, theoretically FP16 might be the fastest, if thats an option in the implementation now.

I did a quick run under profiler and on my AVX2-laptop the slowest part (>50%) was matrix multiplication (sgemm).

In current version of GGML if OpenBLAS is enabled, they convert matrices to FP32 before running sgemm.

If OpenBLAS is disabled, on AVX2 plaftorm they convert FP16 to FP32 on every FMA operation, which even worse (due to repetition). After that, both ggml_vec_dot_f16 and ggml_vec_dot_f32 took first place in profiler.

Source: https://github.com/ggerganov/ggml/blob/master/src/ggml.c#L10...

But I agree, that in theory, and only with AVX512 BF16 (not exactly FP16, but similar) will be fast with VDPBF16PS instruction. Implementation is not there yet.

Interesting.

I saw some discussion on llama.cpp that, theoretically, implementing matmul for each quantization should be much faster since it can skip the conversion. But practically, its actually quite difficult since the various BLAS libraries are so good.

CPU-only, 8-bit quant, Intel Core i7 4770S, 16 GB DDR3 RAM, 10 year old fanless PC: 32 seconds per sampling step, correct output.

Reply on news.ycombinator.com