Hacker News Reader

Comment by speedgoose | Hacker News Reader

Isn’t it just CLIP ? The model that made these image generation models possible.

It’s good to describe a picture but it’s not reverse engineering. The predicted prompt usually has very little in common with the actual prompt. And it’s worse when you use embeddings or fine tuned models.

View on HN

What's interesting to me is that it even tries to predict the prompt on images that came straight from Stable Diffusion with no editing - which is weird because such images actually do have the prompt embedded inside of them already. (At least, that's the case for me - the prompt and parameters are stored in a tEXt chunk in the PNG file, which can be read with, for example, "pngcheck -t".)

True, images generated through some UIs have prompts in meta data, aim here is to work on images people find online with no metadata. So it doesn't try to read the metadata but actually predict a similar prompt

This is something specific to the automatic 1111 version. It's just a setting, but I believe it's on by default.

Not just automatic 1111. Other SD forks like InvokeAI also embed the prompt in the png.

It is based on an image-captioning model, so a different approach then CLIP interrogator, though you are correct that aim is not to get the exact prompt back but actually get a prompt to generate similar styles of images

I'm surprised that the results seem much better (more detailed and sometimes closer to the original prompt) than the regular CLIP interrogation (at least based on my limited experimentation).

But as you say, even so, it still has little in common with the original prompt.

Reply on news.ycombinator.com