Hacker News Reader

Comment by arizen | Hacker News Reader

Are there any publicly accessible Flux image generation services you may recommend to check out?

There are a few.

https://fastflux.ai/ for instant image gen using Schnell (but its fixed on 4 steps and is mainly a tech show off of inference engine by runware.ai)

https://www.segmind.com/ has API support with lots of options, I am using it to generate and set wallpaper using an AHK script. It's very very slow though.

https://replicate.com/black-forest-labs/flux-schnell/example...

https://huggingface.co/spaces/black-forest-labs/FLUX.1-schne...

https://getimg.ai/text-to-image

There are other tools now if you Google 'Flux image generator online'

If you have a suitably powerful M-Mac or iOS device, you can run it locally using "Draw Things", which is free.

Just to clarify for other readers, Draw Things has support and provides download links to quants but no iOS device can run the full precision model which means you will get slightly different and usually lower quality output than stuff you may see elsewhere, even if you use the same settings. It's still damn impressive though.

Quality issue should be mainly due to using FP16 accmulators for GEMM in M1-M2, A14-A16 devices (it is not a problem for SD v1 / SDXL models due to smaller channel count). This is changed to FP32 accmulators for GEMM in these devices with 1.20240820.1 release. q8p should have comparable quality to non-quantized models (in Draw Things, it is called FLUX.1 [dev] (Exact)).

Hey the creator! Thanks for all your hard work. Put in a tip feature so I can give you money.

Claims that quantization doesn’t hurt models are made all the time but rely on the fact that almost all evaluations today of LLMs hardly scratch their surface. If we evaluated LLMs properly, even large quants would be detectably worse, and by a significant amount.

A model trained on BF16 that within the range of FP16 have effective bit rate of 13-bit at max (e5m7). Reasonable quantization (at 8-bit) gets you weight error (i.e. L2 distance on weights) down to < 1e-3, and full diffusion run (30 steps) have result with L2 distance (comparing to unquantized) < 5e-2.

I think there is a line somewhere between 4-bit to 8-bit that will hurt performance (for both diffusion models and LLM). But I doubt the line is between 8-bit to 13-bit.

(Another case in point: you can use generic lossless compression to get model weights from 13bit down to 11bit by just zip exponent and mantissa separately, that suggests the effective bit rate is lower than 13bit on full-precision model).

I buy these kinds of arguments but the moment a company or NeurIPS researcher claims even a "tiny" bit of loss happens, I become suspicious. I don't buy most claims of "we get 99%+ of the performance" made in practice.

But yes, I do believe that we will find proper lossless quants, and eventually (for real this time) get "only a little bit of loss" quants, but I don't think that the current 8 bits are there yet.

Also, quantized models often have worse GPU utilization which harms tokens/s if you have the hardware capable to run the unquantized types. It seems to depend on the quant. SD models seem to get faster when quantized, but LLMs are often slower. Very weird.

I wonder if there is a case for thinking about it in terms of the technical definitions:

If we start from peering into quantization, we can show it is by definition lossy, unless every term had no significant bits past the quantization amount.

so our lower bound must that 0.03% error mentioned above.

I don't think this is true, llama.cpp hobbyists think about this a lot and there's been multiple independent community experiments, including blind testings by a crowd. I doubt it holds across models and modalities, but in llama.cpp's context, Q5 is inarguably unnoticeably different from F32.

However this seems to be model size dependent, ex. Llama 3.1 405B is reported to degrade much quicker under quantization

Not saying I disagree with you. But you don't present any evaluation results for your claim either.

Whoa, thanks for the tip!

Huggingface has quite a few spaces with the base dev model as well as different variations of it.

https://huggingface.co/spaces?sort=trending&search=dev

We're one of the Black Forest Labs partners, you can try the models here: https://www.mystic.ai/black-forest-labs

I made a simple one - https://dreamgenerator.ai. (If you pick the “High” image setting), but u have to subscribe to use it, otherwise it gives you stable diffusion.

Check out https://www.freepik.com/pikaso you can try flux and flux realism

Twitter's image generator is the pro version of Flux.

90%+ of Flux image generations will be done through Grok.

Reply on news.ycombinator.com