For this to be true, we'd had to assume sentience + malevolence. The shortest explanation for why an AI give us correct data currently is a range of confidence. We instructed it to pick the one with the highest number. We instructed it to give us a similar next data etc. We reinforced it, yes, but we'd have to be naive to believe that taking a small part of what makes something "intelligent" and training the shit out of it, then it actually turns intelligent. chatGPT is "just" a large language model, it does not have senses, it cannot move, it cannot live and die. It can only output text. The next gen will output text (maybe) exponentially better. That does not make it "intelligent". But, if we'd combine it with Boston Dynamics, Tesla Autopilot, internet, quantum computing, nano tech and other specifically trained parts (movement, sensing etc) into one, and release it into the world to see what it does, then your argument could (maybe)have a higher confidence of the outcome you fear.