That’s a big claim. If the model was trained on CSAM then I would argue that any image generation that comes from it is harmful to the original children. In the same way copies of already existing content continue to harm the victim.
These models are wildly intuitive and can assemble impossible ideas, even when datasets are purged of certain tags. I discovered this early on when I couldn't diagnose why most subjects came out as weirdly child-like elderly people.
It turns out "mature" is not synonymous with "nsfw" when negative prompting, and a person absent of maturity, while not explicitly a child, doesn't quite look like a normal adult human, either.
I don't think the model was trained on a large corpus of very old babies, yet it was able to imagine them. As long as a model knows what the pieces are or are not, it can be prompted assemble them.
Maybe if it substantially recreates the input images it is equivalent to plain copying and redistribution.
Maybe it creates an economic incentive to abuse more children to create more training data?
And if the used car I buy was stolen, I have contributed to car theft. That's not an argument against used car sales, especially from salesmen that try to make sure their inventory is legitimate: https://arstechnica.com/tech-policy/2024/08/nonprofit-scrubs...