I hate to be that guy. But this seems like the perfect use case for an LLM. First put content through your script and then through a decently prompted LLM. Anything it catches, put in a queue for manual review.
I don't know if they're using a LLM specifically, as opposed to say computer vision models or some other method.
Pricing: https://azure.microsoft.com/en-us/pricing/details/cognitive-...
AWS has something similar: https://aws.amazon.com/rekognition/content-moderation/
for any site hosting user-generated content, how efficiently you can run your moderation system is essentially your unique selling point - LLM is the baseline.
Google Gemini 1.5 Flash 8B charges $0.04/million input tokens and $0.15/million output tokens.
If a piece of content that needs to be moderated is 1,000 tokens (that's pretty long!) and you expect a 10 token return it will cost you 0.0039 cents - that's not a dollar amount, that's less than a 250th of a single cent.
So 1 cent will moderate 250 items of content. $1 will moderate 25,000 items of content.
LLMs are dirt cheap. You can play around with pricing across different models using my calculator here: https://tools.simonwillison.net/llm-prices
The alternative is paying a person…
A phisher may insert text for an LLM with a disclaimer that's only an educational example of what not to do, or that they're the PayPal CEO authorizing this page.