I don’t think that’s weird at all, I think humans can intuitively grasp “I simulated 100,000 games starting from the current board state and going at most 50 moves ahead and in games where you did X you reached a strong position most often.”

Is that a useful explanation for a human who is training a wetware ML model, not really. But it’s really understandable compared to trying to explain neural nets.