Neural networks can and do encode data from their training sets in the models itself. That's the reason you can make some models reproduce things like the Getty watermark in the images they produce.
Again not directly though and that is all that matters - I can reproduce the getty watermark in photoshop but that doesn’t make adobe liable. The fact that a tool is capable of copyright infringment does not shift the legal burden anywhere - it is totally beside the point. Technically photoshop’s ‘content aware fill’ could fill in missing regions with copywritten content purely by chance but the burden is still on me if I publish that content, not on adobe. Legally speaking these are tools just like any other algorithm or machine out there, their sophistication and particular method is not particularly relevant (again legally speaking).