"and we know that it doesn't always work and don't care because despite our ML model being opaque and hard to introspect when we tested it against our database of known bad actors it performed better than both human review and our old heuristic based system."

There is an opposite side to this. If you have a human appeal process your accuracy drops to min(ml_model, humans) and I have bad news about which one of these is smaller.