As it stands AIs today are not always great at making decisions (but they're getting much better), and orgs of today still trust people and hold people to account, rather than their AI systems.
Neither of these are strong moats. It's a moat only while AI systems have some limitations vs an expert human, and corporate processes are still extremely human-centric.
Having accountable people in key positions is a very important part of running a successful organization. Anthropic and OpenAI are never going to let you sue them when an AI employee makes a mistake; accountability is a strong moat.
In the future if you can't trust your AI system to perform a function well, you can switch to another. The accountability will be different – instead of an employee being accountable because their income depends on it, a corporation deploying the AI system be accountable because their success depends on it.
We already see this today with coding. If you're paying too much for the code Claude produces or unhappy with its output, you stop paying for Claude and switch to another.
All will perform roughly at the same level, just like today. It doesn't matter what provider you switch to, they'll all going to make mistakes because performing at a high human level requires far more business context and domain knowledge than is going to fit in even a few million tokens. Humans have incentive to learn and improve, LLMs lack even the ability to improve, as there's been pretty much zero progress on live learning and it's theoretically impossible for a fully-trained (saturated weights) LLM to learn new things without forgetting old things.