Yes, as long as it doesn't disregard the original creator's intentions in the formulation.
And the more meta question is whether the creators' rights should even extend to that realm. Can you specify in a license "This text must not be read by students in the context of an educational course" (which is arguably the closest analogue to AI training)? If that's not possible, then where do we draw the line between tools-assisted learning by humans and human tools learning on their own? You can arrive at answers, but it seems to me like there are many possible reasonable interpretations.
It would be more accurate to compare those students with the corporate interests involved in AI training, not the model's use of the content.
Can we all legally pirate educational books since it's for self training and producing transformative outputs? Can I consume all media (books, movies, music) the same way, and call it fair use? Also for software?
The debate is about content that is generally legally obtained, but might come with certain restrictions. Where restrictions is a broad term and might also just come in the form of a copyleft license, eg. My main point was that in many situations, eg involving open source licenses, it's really not clear from the terms what the creator's intent regarding AI training was. And the broader question is whether training is fair use, or something else, maybe even a new legal concept that would have to be established. Or, what's the difference between art students going to the museum to be inspired, and Dall-E 'looking' at public domain images?
Regarding your point, I wonder how different is it to break the "terms of use" for fair right, vs breaking the "terms of obtaining" alltogether. They both are about jumping over owner's will, who has the full rights of the work.
I very much look forward to see how the ethics and law around these issues evolve.
Otherwise any deluded fool with extreme views on copyright could claim ownership way beyond that which the law currently offers.