I'm outside of my expertise here, but I think if the model is able to reproduce copyrighted text then distributing the model should be a copyright violation.
By your logic, distributing the digits of pi could be construed as copyright infringement otherwise.
of course you can - pi contains all known combinations of digits.
On the other hand, the "address" of a text within a memorizing large language model would just be the prompt "give me the text of X".
The ease with which these "retrieval" operations can be done is irrelevant.