Ask HN: Using Mistral as encoder for SDXL-like net
How would I go about using Mistral 7b as a starting point for making a model like SDXL? Obviously somebody's gonna do it, so I'd like to try myself :) I'd like to use a quantized version, you know?
If you are talking about transformers llms that output pixels, that is an area of active research... I hope you have a few million dollars lying around, though.
I thought that you can put the prompt into Mistral but instead of going all the way to output, you just like take the state of Mistral when it's like halfway done and use that as the "embedding". Then that gives you the prompt embedding or something, then you feed that into the image generator. Is that not how SDXL works?
I don't have a few million lying around, but it shouldn't be hard to get if I have a good prototype that generates 64x64 images you know. Do you know how to make one from scratch?