Anthropic was just fined for not paying for the pirated books they copied into a training database. Same as anyone else who copied pirated IP onto their hard drive.
However, training an AI on copyright has been ruled to be sufficiently transformative and not a violation of IP laws. The same way you can make a gameplay clone of Call of Duty without any issue.