Hacker News Reader

Comment by jrflowers | Hacker News Reader

I hope someday there’s a version of this or something comparable to it that can run on <8gb consumer hardware. The main selling point of Stable Diffusion was its ability to run in that environment.

View on HN

You can do this if you select the `pipe.enable_model_cpu_offload()` option. See this https://huggingface.co/stabilityai/stable-diffusion-xl-base-...

> I hope someday there’s a version of this or something comparable to it that can run on <8gb consumer hardware.

Someday is today: from the official announcement: “SDXL 1.0 should work effectively on consumer GPUs with 8GB VRAM or readily available cloud instances.” https://stability.ai/blog/stable-diffusion-sdxl-1-announceme...

Easy Diffusion (previously cmdr2 UI) can run SDXL in 768x768 in about 7 GB of VRAM. And SDXL 512x512 in about 5 GB of VRAM.

Regular SD can run in less than 2 GB of VRAM with Easy Diffusion.

1. Installation (no dependencies, python etc): https://github.com/easydiffusion/easydiffusion#installation

2. Enable beta to get access to SDXL: https://github.com/easydiffusion/easydiffusion/wiki/The-beta...

3. Use the "Low" VRAM Usage model in the Settings tab.

SDXL 0.9 runs on iPad Pro 8GiB just fine.

Is this using Draw Things, or another app? Did you have to quantize the model first?

Yeah, Draw Things. It will be submitted as soon as SDXL v1.0 weights available. Quantized model should run on iPhones (4GiB / 6GiB models), but we haven't done that yet. So no, these are just typical FP16 weights on iPad.

Thanks! I guess I'll stick to running it on my Macbook for the time being until the quantized model gets uploaded. What kind of performance are you seeing with the FP16 weights on the iPad? I've run a few SD2.0-based (unquantized) models on my 2020 iPad Pro but it seems like it gets thermally throttled after a while.

Will be more info upon release. SDXL v0.9 performs generally the same as SD v1 / v2 on the same resolution. But because you tend to run it at larger resolution, you might feel it slower.

I feel like this is the greatest demand for LLMs at the moment too.

It's hard to believe we're only 8 months into this industry, so I imagine we'll start seeing smaller footprints soon.

8 months from what point?

Gpt3 is 36 months old. Dalle-e is 28 months old. Even StableDiffusion is like 11 months old.

Fair, I should have said 8 months since the market exploded.

We already do. MLC-LLM and Llama.cpp have Vulkan/OpenCL/Metal 3 bit implementations. That can run llama 7B (or maybe even 13b?) in 8GB.

TBH devices just need more ram for coherent output though. Llama 13b and 33b are so much "smarter" and more coherent than 7B with 3 bit quant.

13b q5 llama report „total VRAM used: 8321 MB” so 3bit will most likely fit into 8GB.

Give InvokeAI a try.

https://github.com/invoke-ai/InvokeAI

Edit: Spec required from the documentation

You will need one of the following:

    An NVIDIA-based graphics card with 4 GB or more VRAM memory. 6-8 GB of VRAM is highly recommended for rendering using the Stable Diffusion XL models
    An Apple computer with an M1 chip.
    An AMD-based graphics card with 4GB or more VRAM memory (Linux only), 6-8 GB for XL rendering.

Thanks for the recommendation.

As an aside, does this irritate anyone else?

"You must have Python 3.9 or 3.10 installed on your machine. Earlier or later versions are not supported. Node.js also needs to be installed along with yarn"

I don't like having to install npm when an existing dev stack (python) is already present.

I should clarify that by <8gb I meant “less than 8gb”, which is what SD 1.5 and 2 were able to do. I’m aware that it can run on ==8gb.

There are several papers on 4/8 bit quantization, and a few implementations for Vulkan/CUDA/ROCm compilation.

TBH the UIs people run for SD 1.5 are pretty unoptimized.

Reply on news.ycombinator.com