And humans are trained on the same; unlike AI, we don't have any external supply of art to rely on : ) At any given time, all we have is what we've already created so far
One of the things that is interesting with these new big models is it is dramatically broadening the context in use. The models are learning both the textual representation of a concept, as well as the artistic/visual representation and the relationship between the two domains.
It can generate art out of art. It can do it exquisitely well. But it lacks experiential/social inspiration. This component only comes from recycling here.
(Without the atmosphere of industrializing 19th century Central Europe - we have no Kafka. You can't simply generate Kafkauesqueness out of pre-existing literature.)
And that's why I refuse to believe AI can be creative in a meaningful sense of the word.
Obviously it's not AI's "fault", so to speak, but that's kind of beyond the point :)
PS. I can imagine - now we're going far into the realm of s-f - truly sentient AIs producing art that's genuinely creative. Art actually stemming from a self-conscious AI's psychological experience. But then it would probably be utterly incomprehensible for us : ) "If a lion could speak, we could not understand him", as the philosopher remarked.
Would be very cool to see that experiement.