LLM's response being best prediction of next token arguably isn't that far off from a human motivated to do their best. It's a fallible best effort either way.
And both are very far from the certainty the author seems to demand.
And both are very far from the certainty the author seems to demand.
In my mind the issue is more accountability than concerns about quality. If a person acts in a bizarre way they can be fired and helped in ways that an LLM can never be. When gemini tells a student to kill themselves, we have no recourse beyond trying to implement output filtering, or completely replacing the model with something that likely has the same unpredictable unaccountable behavior.