How does handling support tickets/bugs work with such approach? If LLM sometimes can't handle it - do I have to beg it to keep trying, as humans are no longer an viable fallback?
There's a good chance that if the agent cannot handle it, a human wouldn't be figuring it out either, without additional context. That context would be the sort of only-Joe-knows-how-it-works, so perhaps something worth addressing in any case.