This is a little oblique, but isn't there a parallel to the music piracy debates of the 90s here? The defense of those who steal training data, "It's not stealing because I'm only taking a little tiny piece from a whole bunch of people", seems like a neat inversion of the defense offered by people torrenting mp3s in their dorm rooms.

It's worth noting that, back when it suddenly became technically feasible to copy all songs for free, the response from the rightsholders of music was, essentially, to demand that we all collectively pretend it isn't. It might make sense for the rightsholders of training data to try to do the same, if they could speak with one voice and if they had lobbyists, but I guess the fact that they can't and don't settles that.