As an aside, I wonder when llm’s will be weaponized against authoritarian (China, etc) human censors, as they could be totally overwhelmed with content generation and be forced to censor all internet traffic…
Spam sites are still spam sites. The only exception are sites like CNET co-opting the tools to seed AI in combination with their own real bloggers (saying 'journalist' would be a stretch here).
These still require reputable people/business to be behind them directly to give credibility, people that can be blacklisted. It's not like turning an AI bot loose and instantly you have 1000x fake Twitter accounts with 100k followers and top ranked on Google.
The "long tail" of spam sites do not threaten governments.
China cares most about reputable news sites and popular social media influencers going viral with taboo stories. I'm skeptical news sites or influencers can be generated artificially merely using AI, in a way that is scalable enough to actually be a problem for censors. Ultimately the censors just need to ban the source... the domains or accounts with 20-100k+ followers or news sites with actual reach (ultimately a small set of sources). Fake AI content doesn't directly solve the existing reputation signals.
The more I think about it the less I'm convinced this is a real threat to the general reputation networks.
That said - where this does matter is the typical grey/black market where scams already operate. Not in highly public reputable places like the top rankings on Google or Twitter or w/e popular Chinese social network. It's the hackers exploiting email lists, Indian call centers running combination IM/phone scams, etc. It's the low-end of the market for criminal get-rich-quick schemes, not something that generates popular movements that threaten governments or popular political ideologies.
The idea that censorship is a weapon of authority is a pre information age idea when signal-to-noise was high.
Everybody, as far as anyone could tell, in complete agreement with the regime, all the time.
Signing and encryption will come in useful. But not sure of much good it will do under an authoritarian regime.