So much of modern technology is a trojan horse these days. This is basically the enshittification of cloud services. Just a few years ago people would say you were being a little paranoid if you were worried about your data passing through company's servers unencrypted, but here we are now.

If companies do start training models on what people consider to be private documents, then the issues we already have with AI taking the jobs and purposes of humans is going to become significantly worse.

Scientists working on papers will essentially not be able to trust that their work won't get out before they have published it. A competing research team, asking prompts in the right way, could chance upon a reply that gives them a clue as to what the other team have found or are doing. The same goes for competing companies and engineering teams. Or authors writing the next book in a series. Other people using that trained data could produce a cheap rip-off of that next paper, patent, book using AI.

And that will completely demotivate humans to actually do stuff. Because what's the point? No one will pay you for it, and a poor quality second rate product is obtainable much cheaper.

At that point I think we'll discover what the real limitations of AI are, as we, as a society, have to get used to using it over humans. And I somehow doubt we will be better off.