The current discussion seems to fall under "Trust and Safety". I'm not aware of any specific guidance or organised thought, though I'd be
really surprised if there weren't academic coursework or professional training beginning to appear.
You'd do well to look at established services. Craigslist's list of prohibited content, the T&C of Facebook, Twitter, Reddit, etc., are going to be useful.
Just off the top of my head:
Cyberstalking, pornography, child porn, piracy, malware, fraud, illicit goods (guns, drugs, black/grey market, stolen goods), intimidation, gangs, bullying, alcohol, tobacco, prescription medications, hoaxes, various ineefective / "alternative" products and remedies (which themselves run the gamut of legality, even defining this is at best difficult), advertising, advertising for protected or regulated sectors / goods (housing, employent, personal and professional services, beauty care, escorts, security services, licensed professional, ... As with the goods section, this rapidly gets complex), legal services / aid, political activities, fomenting revolutin / freedom fighters.
User-generate content is a massive concern.
One concept I'm seeing getting increased traction is a focus not on the quantity of posted content but the prevelance or level of access or views. Facebook and YouTube especially are increasingly discussing problematic content not in terms of posts or videos, but of views or presentations of those.
This ... starts making trade-offs in moderation much more viable, principally because there is an inverse logarithmic relationship between the number of items and the views: If n items gets n views, then 10*m items get n/10 views. Very roughly.
This means that you can set a goal in terms of the number of items viewed (and see what the maximum unmoderated prevalence will be), or target a specific prevalence and determine how many reviewers will be required.
For human moderators, the number of items reviewed per day seems to be in the 500--800 range. Note that 800 items/day in 8 hours is 100/hour, or 1.6 per minute, or 36 seconds per item. That's inclusive of breaks, overhead, and non-moderation tasks.
Moderation itself is a very psychologically loaded task. You'll either want to rotate people through it from other functions, or see a heck of a lot of staff turnover.
If anyone has greater insights from one of the current large UGC services (FB, Twitter, Instagram, Whatsapp, TikTok, Imgur, Reddit, etc.), I'd really like to know what current internal practices are.
Some of my previous work had some incidental exposure to this area (I was tasked with removing identified content, working on both our internal and external CDN provider to do so). After a couple of spot checks to see I was unlikely to be deleting content which wouldn't meet removal criteria, as in literally two, I decided I simply didn't want to take the risks of performing additional checks. My removal process turned out to be quite effective --- what the CDN provider's specs suggested might be a weeks-long process removed some millions of items over a weekend. That was on what is by current standards a very modest-sized social network.
I've written on this previously citing YouTube and Facebook sources here: https://joindiaspora.com/posts/f3617c90793101396840002590d8e...