I haven't really been following the AI stuff that closely, but in a similar conversation last week someone was noting that while a lot of the model creators said the original images aren't there, in some/many cases they actually are and if you ask correctly the exact (or close to it) image comes out the other end.
I don't know the details, or enough to have a real opinion on it, but it seems like if you have a system that you can request something from and an input image comes out mostly intact on the other end, that shares a lot of similarities to a database and lossy compression. If that's actually possibly, then I have questions about how much of an original is used in the output even if it's changed, and at what point it becomes transformative.
Again, I'm not sure about most of this, and don't even have the source comment I'm remember, much less their source for their assertions, but I do have lots of questions and suspicions, if indeed what we think we know about these models based on prior statements ends up being wrong in some cases as we learn more.