If it's going on in any large scale way on a major site, I've not encountered it (but it might not be something they'd shout about) but similar types of models are being put into apps frequently, as seen in this post recently posted to HN: https://altayakkus.substack.com/p/you-wouldnt-download-an-ai (a mobile app that detects currency client-side)

One problem is legal liability around training and testing. If you use a large foundation vision model, you could describe your moderation criteria in text and cross your fingers, but those aren't going to be running client-side. If you want a compact, efficient model to embed into an app, you need to train it.. but with what? There's a dilemma here where it's easier to build filters for legal things because you can't legally use the illegal things to train a filter for the illegal things! Systems like NSFWJS work because run of the mill porn is legal in most jurisdictions, but an equivalent "CSAMJS" filter would be a legal nightmare to produce.