Hacker News Reader

Core takeaways:

- major improvement is from detailed image captioning

- they trained an image captioning model to produce short and detailed captions

- they use T5 text encoder

- they use GPT-4 to "upsample" short user prompts

- they train a U-net decoder and distill it to 2 denoising steps

- text rendering is still unreliable, they believe the model has hard time mapping word tokens into letters in image

They have only distilled the diffusion model used as a replacement for the VAE decoder, not the main diffusion model.

So the latent to pixel space conversion is made with another diffusion model?

This is kind of refiner used in SDXL?

>So the latent to pixel space conversion is made with another diffusion model?

Yup

>This is kind of refiner used in SDXL?

The refiner only works on the latent.