> But there's no way to train the transformer-based high-probability-next-word AIs to be superhumanly good at fooling you into doing something, on the grounds of lack of training data

The conversations of all those human scammers would be prefect training data for this. You even know exactly what conversations led to payouts. Assuming you can get all your data in one place, of course.