>I don't think people would obey an AI oracle in that way

You clearly haven't encountered the large group of people who trust hallucinated LLM output more than their coworkers.