I can see how this would work if you just turned off your brain and just thought of course this will work
https://cybersecuritynews.com/fraudgpt-new-black-hat-ai-tool...
If you make an app for interacting with an LLM and in the app the user has access to all sorts of stolen databases, and other conveniences for black hats, then you've got what was described above. Or I'm missing something?