It's not that the watermark is on them per se, but that the model tried to emulate an image it had seen before which had a watermark on it. Imagine showing a child a bunch of pictures with Getty watermarks on them, then they draw their own, with their own emulation of the watermark. They don't know it's a watermark, they don't know what a watermark is, they just see this shape on a lot of pictures and put it on their own. That's essentially what's going on.

The model is only around 4GB, and it was trained on ~5B images. At 24 bit depth, that'd be 786k raw data per image, which would be 3.5 petabytes of uncompressed information in the full training set. Either the authors have invented the world's greatest compression algorithm, or the original image data isn't actually in the model.

So, I think the argument is: if you look at someone else's (copyrighted) work, and produce your own work incorporating style, composition, etc elements which you learned from their work, are you engaged in copyright infringement? IANAL but I think the answer is "no" - you would have to try to reproduce the actual work to be engaging in copyright infringement, and not only do these models not do that, it would be extremely hard to get them to do so without feeding them the actual copyrighted work as an input to the inference procedure.