It’s one thing to ask an algorithm how to build an A* driving map from point A to point B. It’s another to ask one how to be a better person and go to Heaven.
I’m not religious, and I’m not arguing this from a pro-religion POV. I happily work in AI, and I’m not arguing this from an anti-AI POV. I am highly technical. I love computers. I’m excited about the future. I rely on deterministic algorithms to make my days better. And yet, I do not want to trust the words of an LLM to counsel me on how to be a better husband or father. At this stage, the AI does not know me in the way a counselor or advisor, or even pastor or priest would. And yes, I think that’s a crucial difference.