So the best experience was this internal event that we had, where the raters would say that certain Ad would not fare well (long term), while the initial metrics (automated) were showing the opposite (short temr). So then we'll gather into this event, and people would "debug" these and try to find where the differences are coming through.
Then we had to help another group, where ML failed miserably detecting ads that should've not been shown on specific media, and raters came to help giving the correct answers.
The one thing that I've learned is that humans are not going to be replaced any time soon by AI, and I've been telling my folks, friends or anyone (new-born luddities) - that automation is not going to fully replace us. We'll still be needed as teachers, evaluators, fixers, tweakers/hackers - e.g. someone saying - this is right, and this is not, this needs adjustment, etc. (to the machine, ai, etc.).
Maybe machines are going to take over us one day, but until then, I'm not worried...
(I've also understood I knew nothing about staticics, and how valuable linguists are when comes to forming clear, concise and non-confusing (no double meaning) questions)