Segmind Stable Diffusion – A smaller version of Stable Diffusion XL
huggingface.co
huggingface.co
SDXL isn't there yet but we're getting mighty close. SD1 felt a lot like shouting words at an idiot savant to try and inspire something good. SDXL gives me the impression I'm explaining what I want to a talented but verbally challenged painter who will attempt to draw what I want. The trajectory seems to be that we'll reach something close to natural English in maybe 2-6 years and if AMD could please keep fixing crashes that'd be lovely.
But seriously, halving the model size and doubling the speed will mean a lot more people can use it. That is a big deal because some of them will spend time training in more data for specific applications and setting up tooling to work around limitations.
SD 1.5 has a ton of inertia, and ultimately its still smaller than SDXL.
I love that LLM are getting more accessible and I want that trend to continue.
'list' object cannot be interpreted as an integer
(BTW, regarding the error you saw - use this instead https://huggingface.co/spaces/segmind/Segmind-Stable-Diffusi... )
A lot of that is using an LLM for prompt expansion and composition guidance, both of which can be done with SD just by looping in an LLM for those purposes, though there isn’t support in any of the popular UIs for LLM composition assiatance (there are recent papers denonstrating it, though, so I wouldn't be surprised to see extensions for popular UIs within a month.)
EDIT: corrected to clarify that only composition assistance isn’t yet supported in popular UIs, because prompt expansion is built in to Fooocus.
At a high level, the SD architecture looks like this:
noisy latent -. +----------+
+------+ `-> | UNet | +-----+
prompt -> | CLIP | -> embedding -> | diffuser | -> denoised latent -> | VAE | -> output image
+------+ +----------+ +-----+
SDXL's better text encoder is partially why it can understand sentences better than SD 1.5, and you can swap it out for, say, BLIP for combined image + text prompting.
The huge leap mostly comes from text descriptions that aren't trash.
My impression is there's a large open development community that has come up with some really clever stuff - fine-tuning, LoRAs, ControlNets, regional prompting, and so on. You can teach the model completely new concepts and styles at home, for <$1 in electricity.
But it's all been built on top of a base model trained by Stability AI at a cost of $600k (or at least, it would have cost that at AWS GPU prices).
And some imperfections like mangled hands, prompt concept bleed, the inability to generate clock faces and mirrors etc seem to go deep into the base model, and beyond the point that fine-tuning can fix. And there aren't many people with $600k of spare cash lying around to train a new base model.
The outputs of the open source models are great despite such weaknesses IMHO - but the costs involved mean the open source community is fighting with one hand tied behind its back.
Of course, open models will always win in some areas - like if you want to generate a picture of Xi Jinping.
The LAION dataset they used was notoriously poor quality, and the training process wasn't optimal by any measure, though. The costs of training are rapidly falling due to various optimizations.
Take a look at Pixart-alpha [0]. They claim SDXL-comparable performance for just $26k in training from scratch, with just 600M parameters in the unet and 25M pictures in the training set. Supposedly they achieved this due to the high quality training set tagged by a third-party model. The weights got leaked recently and the claim looks believable.
Unfortunately, according to their Readme, "Inference requires at least 23GB of GPU memory." So not something you can run on consumer hardware in its current state.
But try it in Fooocus, with its sea of augmentations (including Dall E-like prompt expansion): https://github.com/lllyasviel/Fooocus
I have stuck with 1.5 because SDXL wasn't that exciting to me and ultimately I haven't been clear what software needs to change (and how/why) to be "compatible"
Its extremely similar, but not drop-in compatible, no.
Whats more, small changes make it compatible, but larger changes are needed for SDXL to work well.
Personally I find SDXL to be a huge upgrade, at the cost of performance, hence I am using Fooocus but keeping an eye on https://github.com/FizzleDorf/ComfyUI-AIT
As I have been preaching on HN, just don't use A1111 for SDXL.
Why, what am I missing by doing so?
Less jankiness.
For me, personally, better hackability, especially in `diffusers` UIs (though Comfy-based UIs seem to be SOTA for SDXL).
EDIT: Also, other user interfaces are more "suited" to SDXL and its presets+style prompting. A1111 is very much built for SD 1.5
And yeah, the difference is dramatic!
I just had to update A1111, download the sdxl model, download the sdxl refiner model and set the refiner in the refiner dropdown. That's it. So compared to other models it is just one additional dropdown click.
You won't be able to use the LoRAs, etc. you are used to, but you can download new ones that are compatible with SDXL.
You should be using Fooocus (which IMO has the best out-of-the-box quality), InvokeAI or ComfyUI for it instead.
> oobabooga rolls out support for new llama models
ooba is kinda different than a1111 under the hood. Its a big bundle of different LLM backends with as much common code hacked in as possible, while a1111 is the single, original SAI backend with a ton of features globbed on.
Technically ooba doesn't have to "support" any new models at a low level, thats typically the job of the backends. What it adds are new features and integrations.
Fooocus is new and seems focussed more on casual use compared to the other three.
Its, by a mile, the best SDXL output I've gotten from any UI, even with lots of tweaking.
A1111 indeed has tons of tools built into the UI through extensions, but they are mostly focused on SD 1.5.
ComfyUI and Invoke bundle lots of tools too, but they are "hidden" behind the nodes UIs and sometimes less functional.
IME, most of them are agnostic, because most of them wrap around rather than hooking into the inference process.