OpenAI releases Consistency Model for one-step generation

This is a very significant paper. The first diffusion paper required 1,000 steps to generate an image. By last November, they were down to 50 steps. This paper takes us to one or two steps. What we're talking about here is nothing short of magic. You give random noise to a computer and it generates a realistic image of whatever you want in just one or two iterations.

Correct me if I'm wrong, but I didn't see anything about training time or cost. I would be interested to know whether it is more or less expensive to train this model than it is to train Stable Diffusion.

1. Consistency models are a new type of generative models designed specifically for efficient one-step or few-step generation. They achieve high sample quality without adversarial training. 2. Consistency models can be trained in two ways: (1) Consistency distillation: distilling a pretrained diffusion model into a consistency model. This results in high quality one-step generation. (2) As a standalone generative model without relying on a pretrained diffusion model. This still achieves strong performance for one-step generation, outperforming other non-adversarial single-step generative models. 3. Consistency models allow trading off compute for sample quality by using multistep generation, similar to diffusion models. They also enable zero-shot image editing applications like diffusion models. 4. Empirically, consistency distillation outperforms existing distillation techniques for diffusion models like progressive distillation, achieving state-of-the-art FID scores on CIFAR-10, ImageNet 64x64, and LSUN 256x256 for one-step and multi-step generation. 5. As standalone generative models, consistency models outperform other single-step, non-adversarial generative models on CIFAR-10, ImageNet 64x64, and LSUN 256x256, though not GANs. 6. Consistency models share similarities with techniques in deep Q-learning and momentum-based contrastive learning, indicating potential for cross-pollination of ideas. 7. Some limitations and future work include: - Evaluating consistency models on other modalities like audio and video. - Exploring connections to deep Q-learning and contrastive learning in more depth. - Developing more sophisticated training methods for consistency models. - Improving the efficiency and stability of the multistep sampling procedure.

I guess in a way, OAI chose the right approach in neglecting Dall-E and basically left the scene to Midjourney and Stable Diffusion. I think with their capability and resources, OAI can easily make something competitive to the current Midjourney and SD but they instead focused on the bigger picture to better fit their capabilities.

Small models on the scale of a few billions parameters is probably best left to crowd sourcing efforts because almost everyone can do it. Dealing with large models like GPT and engineering entirely new approaches like this one is more expensive and not easily done by the community. So I guess it is the most efficient use of resources on both side.

tl;dr, a faster alternative to diffusion models for image and A/V generation.

Abstract of the paper:

> Diffusion models have made significant breakthroughs in image, audio, and video generation, but they depend on an iterative generation process that causes slow sampling speed and caps their potential for real-time applications. To overcome this limitation, we propose consistency models, a new family of generative models that achieve high sample quality without adversarial training. They support fast one-step generation by design, while still allowing for few-step sampling to trade compute for sample quality. They also support zero-shot data editing, like image inpainting, colorization, and super-resolution, without requiring explicit training on these tasks. Consistency models can be trained either as a way to distill pre-trained diffusion models, or as standalone generative models. Through extensive experiments, we demonstrate that they outperform existing distillation techniques for diffusion models in one- and few-step generation. For example, we achieve the new state-of-the-art FID of 3.55 on CIFAR-10 and 6.20 on ImageNet 64x64 for one-step generation. When trained as standalone generative models, consistency models also outperform single-step, non-adversarial generative models on standard benchmarks like CIFAR-10, ImageNet 64x64 and LSUN 256x256.

https://arxiv.org/abs/2303.01469

This looks like important research, but the pre-trained models may not do what you hope. From the model card [1]:

> These models sometimes produce highly unrealistic outputs, particularly when generating images containing human faces. This may stem from ImageNet's emphasis on non-human objects.

I guess we’ll see what other companies do with this research? It would be great to have image generation times that are closer to image search.

[1] https://github.com/openai/consistency_models/blob/main/model...

Can anyone ELI5 how the hell this is possible? I think I have a decent model of how diffusion models work: They're trained by teaching them to denoise a bunch of images, so they eventually learn how to create images out of noise and the prompt biases them as to what they should "see" in the noise. But this just seems like absolute magic.

Are they going to make something like a Dalle-3 based on this technology that is trained on like, every picture, and runs on their giant GPU farms? I think this academic paper only uses relatively tiny research data sets to compare with other methods.

Explain this to me like I don’t know this subject