What happens if we train a neural network on a single, copyrighted work? Say it has one input node (or even zero, if you like), and regardless of this input, its output is always exactly the copyrighted work it was trained on. What do its weights represent? Clearly, its weights represent a direct encoding of the original work. Those weights are copyrightable, but not by the person who trained the neural network -- the copyright is held by the owner of the original work.
What if we train the neural network on just two copyrighted works? If its one input node is 0, it outputs the first, and if it's 1, it outputs the 2nd. Almost certainly, its weights are a complicated, tangled mix encoding both, like a compression algorithm that completely rearranged its input. Who owns the copyright to those weights? To whatever extent the weights can be "factored out" into a set representing the first work and a set representing the second, clearly the copyright holder of the first work holds the copyright on the first "factored set", and the 2nd on the 2nd. It seems obvious that we must be able to do this "factoring out" somehow (even if the topology of the factored networks is different), because we know both works are exactly represented by the weights, and the neural network itself can use this information to reconstruct them both, so they're in there ... somewhere. So is there a sort of "joint copyright" on the combined weights, where nobody is really allowed to do anything with it without approval of the other? Regardless, it's still clear that whoever trained the neural network has no claim on any copyright.
Where is the breaking point extending this from 2 works to a billion? People make arguments like "drawing a car from memory isn't infringing on copyright design of that car", which ... are you sure? Reproducing a piece of music from memory (and selling it) is usually copyright infringement. You're allowed to learn a Taylor Swift song as part of your musical training, but you're not usually allowed to then play it back from memory and sell that recording (I'm not sure I morally agree with this treatment of covers, nor if it's globally applicable). So the argument that "surely neural networks are allowed to learn from copyrighted works" misses the point: they can learn all they want, but as soon as they reproduce verbatim (or close enough) a copyrighted work, they're infringing. And if they're representing a complete copy of the work within their weights (which they obviously are if they can reproduce it), then the original copyright holder has a claim on those weights. And never in this process has the trainer of the NN acquired any copyright to anything. The real trainer is a bunch of GPUs, after all.
If the neural network cannot reproduce any of the copyrighted works verbatim, then we're getting closer to "fair use" territory. Yes, it's permissible to write a summary of a copyrighted work. That is so lossy as to not "compete" with the original work in any meaningful way. If it could be demonstrated that neural networks do not encode completed works (no matter how hard the factorization would be), then one could make this argument. Unfortunately, the evidence is that LLMs are more than happy to completely regurgitate copyrighted works verbatim. It seems to me the copyright holder of the original work therefore must hold a share of the claim on the weights. Still, the GPUs that trained the network do not magically acquire copyright over anything.
I wonder if the real answer is that the weights are copyrighted, and that copyright is held jointly by hundreds of millions of people, and nobody can do anything with those weights without the approval of all the others. I'm not saying I like that universe, but I am saying it's the most internally consistent answer I can think of, and seems to follow from the above argument.