The possibilities made available for bad actors to manipulate the masses with this technology is unprecedented and terrible.
I think there needs to be a return to a more siloed, community based, web-of-trust model of communication where there is confidence that the people being interacted with are actually human.
A persuasive, funny, distributed army of commenters that sound like real people that are given prompts by people with the resources to spin up accounts undetected (or allowed via backdoor deals) and mimic the general public is nightmare material. I think a fair bit of that kind of manipulation is already starting to ramp up.
This technology is in my opinion on the same scale of danger as nuclear weapons and needs to be treated as such. It’s insanely dangerous.
I don’t think it can be regulated out of existence, and that also risks concentrating it in the hands of bad actors. I think attempts to regulate it effectively should still be made. But I think the only practical way out of this is some kind of distributed private set of communication networks where people control their own servers, their own online identities, and only connect to people they meet in real life (and then connect to others through networks of relations). I think that’s more realistically accomplishable then it sounds and is desperately needed.