It’s not derivative work though. First, a human didn’t create it, so copyright protections don’t exist on its output. Machines don’t enjoy copyright protections, people do.
It’s mechanically copying and reproducing part of its input data set. Making a tool that regurgitates others’ copyrighted IP is going to been seen as aiding mass copyright violations. Exactly like how Napster got sued: they’re holding a bunch of material they shouldn’t be. The only new twist to this case is the data is encoded in a transformer’s weights. This should be correctly seen as the same as having encrypted copyrighted data using a lossy algorithm.