Indeed. A difference I see between the GPT-like language models and those image generation models is that it feels like the language models actually hold full copies of a lot of sample of the training dataset (hence their ability to recite existing content), whereas the image generation models clearly do not: Stable Diffusion is 4Gb something, yet it can draw anything.

That's the amazing part: the training dataset contains 5B images. Yet it distilled all of them in this mere 4Gb of data and can produce an infinite amount of content with those. It really feel like it learned how to draw, in the same sense a human learn and does not simply reproduce exact copy of what it already saw.