Ask HN: Tools detecting malicious/non-legitimate content

Hey everyone, I’ve been building something over the past few months on this space and would like to understand how platform creators/managers are solving this problem right now.

If you are building (or involved in) a platform with user-generated content and have (or would like to have) implemented something to detect of malicious/non-legitimate content, I’d like to know what it is and what’s your experience with it (or what would you like to use and why).

Right now I have solutions working for malicious links and non-legitimate emails, but am open to exploring more (NSFW image, video, and audio, or full message content, for example).

I hope this isn’t breaking any rules and if you think this is too sensitive to share publicly, I’m happy to talk via email (you can find it in my website, which is in my profile).

Thank you!