And I do agree with him: If whether you get work or not decides on a black box AI and your only support channel is another black box AI, that is pretty much a Kafka-esque nightmare.
I'd guess we here on HN are the people best equipped to prevent the worst. So the next time you help someone automate their customer support, ask yourself: How would I feel if my well-being depended on this? Because for some poor soul, it might. Is there a clear fall-back in case the AI fails horrible? Because, you know, they always do.
I once had my own support automation problem with Amazon, but luckily I had no stakes in it. They accidentally sent me someone else's parcel. So I filed a support request to inform them. They very politely apologized within a minute and informed me that they are sorry about my lost parcel and they'll send another one. So I got the same wrong parcel again. After waiting a while, I opened up the two parcels, each roughly 10x5x5 inches (25x10x10 cm) large. It was two single pencil erasers.
But boy would I have been furious if I had received the same support quality for a lost high-value parcel... Also, I did ponder if it is OK for Amazon to waste my time if I'm not even their customer. I mean their support forms are difficult to reach, no matter why you need to contact them.