Hate Chatbots? You Aren't the Only One
wsj.com
wsj.com
They seem optimized for situation C) where I need something that is easily understood/available, but am actually lazy to look it up, which doesn't really happen.
And so, my typical chatbot interaction is me trying to speedrun through the bullshit in order to get to actual help.
There was one notable exception, where the chatbot was actually there to ask 2-3 questions/give informations to prepare for the following human interaction, and made it in such a way that user saw the goal. I wish all chatbots were like that.
My understanding from talking to a couple of CS execs is that these have been a slam dunk in terms of ROI because CS agents don’t need to handle type C requests. I expect we’ll only see more as time goes on.
But I once managed to get through to an actual agent with this question:
1. I want to buy a kindle version of this book [amazon link, for the paper version of the book].
2. On the page for the book, there is a link for the kindle edition: [link].
3. That link goes to a page for what appears to be an entirely different book. (Under the same name; this was an edition of the Arabian Nights.)
4. However, I have independently found this page: [link], which appears to be for the kindle version of the book I'm interested in.
5. Given that I want to buy the kindle version of the book linked up in step (1), which one should I purchase?
The agent directed me to buy the book that purported to be the book I wanted, instead of the book that Amazon believed was the book I wanted but which claimed to be something different. I would have assumed that anyway. But a couple days later I checked on the book and the "kindle version" link for the paper version had been corrected.
Unfortunately, while they did correct the issue on the one book that I took the time to point out to them, it's still rampant all over their website.
Such as:
"Is your device turned on?"
"Are you logged into the site and not just searching google for the thing your want our application to do?"
"Have you actually purchased our product and not a competitor's you just think is similar?"
Pharmacy Robot: Hello, thanks for calling <pharmacy>. What can I do for you? You can say anything like, "Check pharmacy hours" or "order a refill".
Me: Hi, I have a refill for <specific medication with rules around it> that is due next week but I'll be traveling out of the country to <other country> for a couple of weeks. I need to know what my options are.
Pharmacy Robot: Ok, you want a refill. Please enter the prescription number now.
Me: No, if we try to refill it, the automated system will just reject it. I need to talk to a h...<cut off by robot>
Me: Jesus Christ, do I have to hang up and go through this whole thing ag... <cut off by robot>
Pharmacy Robot: Sorry, I didn't get that number. Using...<cut off by human hanging up>
That's just the most recent one I had. There are often better examples of madness...
That's not correct. I NEVER call without first exhausting every available source because I despise the phone system and it's inefficiencies. Most companies may think they have resources available, but they really don't. And no, just throwing up a zendesk or equivalent "knowledge base" isn't the same as providing tools and manuals/guides/etc.
That said, there is definitely a subset of people for whom calling is step 1 (before even googling). They tend to be older and/or on the tech illiterate side. But if you design and build for the worst-case scenario, you're really screwing over your more self-help customers and even driving them away.
This was before LLM stuff.
Funny anecdote. The most common question was "what is the price/how much does it cost." Which always was strange to me since if they were on the website with the chatbot the price was right there.
There was an article a while back that said something like most books sell less than 300 copies. I think the real numbers turned out to be higher, but still really low. Reading is not a popular activity. Watching video is popular. Talking to people is popular. These are skills that children acquire naturally without effort. Reading: difficult, slow, isolating. Given a choice between talking to someone or reading, people will go through a lot of pain for the opportunity to talk.
This was one of the hardest lessons to learn in my software career. I still suck at it. It's one of the lowest hanging fruits for me to improve as a developer. Just today I was shortening a popup message that my company's software sometimes puts on the screen [1] at the suggestion of a customer who had to deal with support requests from people who weren't reading it. Kicking myself!
Another popup this software rarely has to generate says "Install failed. The following error has been copied to your clipboard .... <stuff>". 100% of error reports, and I do mean 100% without exception for over two years now, come in the form of screenshots. Exactly zero people have ever read the second sentence and used paste. Kicking myself again! Why did I even think that would work?!
The hardest thing is developer docs. I still don't know how to strike the right balance there. We should make more videos.
[1] Well actually Sparkle.framework, but Conveyor incorporates it. It makes distributing desktop apps super easy, check out bio, it's cool.
Could be an electrician or something, driving the car from one customer to the next it's easy to wait in a phone queue and then ask someone for the information. Relative to sitting down and being useless for a while, thumbing away at some badly structured corporate FAQ, that can seem quite attractive. When you're on your break, you're on your break, you want to savour the cinnamon bun and coffee, and would rather not sully it with some boring web page.
then again, misanthropy is likely the core sales pitch with digitization
I did phone support for pretty advanced technology things (you had to be pretty involved in IT to even have one of these) and still a decent number were fixed by "read the FAQ, do steps 1 and 2."
Yes, exactly.
And I HATE calling phone numbers for something I need. By the time I'm using the phone to call you, I've already exhausted every possible avenue I have, and I'm also already pretty irritated. At the least, make it so your chatbot tells the human what I've already said so I don't have to repeat myself once I get a human. Also, if the chatbot doesn't understand what I'm saying (which is nearly always), just send me to a human.
I think it's utterly insane that things have gotten so bad that now Google Pixel phones come with a system to handle that automated crap for you. We're now building bots to talk to the bots? I'm extremely grateful for the Google product, don't get me wrong, but the fact that it needs to exist is in my opinion a great example of how terrible things are with phone support.
Sometimes they can serve as a useful intake system for collecting information before passing someone off to an agent, but even then they tend to be less useful than just filling in a form. I think in most cases, a step by step form followed by a live chat would be better.
Unfortunately there is a segment of the population that is completely incapable of filling in a form. These people should probably be passed off to human beings, but then those human beings are wasting their time dealing with similar issues constantly.
LLM chatbots are even worse. They have great latitude to say convincing and incorrect information, but can rarely actually do anything. It would be much better to just publish a good knowledge base.
I would say a pet peeve of mine is when you do end up having support staff that are human beings, but are not empowered/trained to actually solve customer problems sufficiently. From a business perspective a chatbot is a great replacement for that, because it also regurgitates a script without actually having to action any business processes.
The problem of course is that users have all been trained that knowledge bases are terrible and will go straight to support without even bothering to look.
Many users also don’t really want to become an expert in your thing. They just want someone to solve their problem and don’t care how
For example: I could learn the ins and outs of quickbooks, but I don’t really wanna. The once or twice per year that the UI won’t let me easily do what I want – straight to support.
I work in accessibility.
Most of the forms I've worked with are horrendously bad for accessibility. Chat bots are even worse. People with vision or cognitive issues are an afterthought these days. As we've forced more and more people to use these poorly coded or poorly designed applications and moved away from allowing people to talk to an actual person, we are creating an internet that is increasingly less accessible to people who really need it.
The sad thing is way too many devs I work with have no idea how to write semantic HTML any more. Just simple things like putting things in an unordered list seems beyond some of the frameworks and CMS they use.
It takes a ton of review, SQA, and dev time to get this stuff right. Writing semantic HTML is just the tip of the iceberg. Nobody is an expert on it either no matter how well they've read WCAG and how much experience they have implementing it. It's a moving target and every page or app is different especially when there are any design or functionality changes. It's a team effort. You can't just dump all responsibility on devs.
I would be a strong advocate for some regulatory framework requiring an effective right to access human support for businesses exceeding a certain size.
> LLM chatbots are even worse. They have great latitude to say convincing and incorrect information, but can rarely actually do anything. It would be much better to just publish a good knowledge base.
Just as there is a segment of the population that is incapable of filling in a form, there is a (probably larger) segment which is incapable or at least unwilling to educate themselves with a good knowledge base. There are also a lot of bad knowledgebases with incomplete, obsolete, or misleading information out there, which doesn't help the cause.
Yes, but a lot of companies seem to institutionally incapable of doing so. The number of FAQ pages (or worse, FAQ apps) I've come across filled with questions that no human has ever asked about a product, but instead list all the questions that the marketing department wished people would ask about their product because the answers paint it in such a good light, is astonishing.
The trouble with an actual knowledgebase that helps people who are struggling to use your product, is that it broadcasts to the world the ways in which your product is hard to use. And that just doesn't look good. So that info is either not published at all, or made deliberately hard to find, in order that people can't figure out where the warts are.
A phenomenal amount seem to collect useful identifying information, and then the agent ... asks you for the same information again? I don't know why.
How could you remove the generative part from an LLM? (genuine question)
Each time I asked the tool a question, it responded with completely unrelated garbage, in one case even spitting out fake tracking info for an order I never made. This looks SO BAD, I'm truly baffled that any company has signed on to use such a service.
Rachio has replaced their customer service with an LLM that can only link you help pages. 100% useless. If I hadn't been able to find their real phone number from Google I wouldn't have been able to use their device.
Really made me question buying anything from them
With time best practices and common patterns will get established.
As a user, most of my interactions with pre-ChatGPT breed of chatbots (including "voice bots" on the phone) involved the robot having the capacity to help me at least partially, but failing to understand my request. Fixing that doesn't even require accurate tool use - it requires using LLM on input to parse it in place of whatever they're currently using, instead of directly as a chatbot; something even pre-GPT-3.5 completion models were somewhat good at.
I'm all for giving people time with new technologies, but here, 1) somehow they managed to take what should be an out-of-the-box improvement and use it to make things worse, and 2) myself and people like me are on the receiving end of the problem. At scale, there's lots of extra real frustration being created, and lots of additional people-years wasted, through companies jumping on such tools to potentially save a buck by automating away a few jobs. Hell, it probably shows as healthy profit on the books, as it just further disempowers customers, who learn to take a beating instead of fighting, because what's the point anymore...
As tool use becomes more widespread and context window sizes go up, I bet we'll start to see blog posts where devs talk about how they made your approach work well, and then others will see it makes sense and start to copy it.
It's like some people have lodged themselves into this niche and want to bullshit some kind of demand for it into existence
However, one way in which chatbots are often preferred: They cost less to the business. Whether they're good or not seems to be a secondary concern. So, if your coworker is describing what the people inside the business want, they're probably right!
I'm baffled, which scenarios?
An example might be:
There's something wrong with my order -> wrong size -> order not shipped -> update size on order to small -> small out of stock -> cancel or backorder
or
There's something wrong with my order -> wrong size -> sorry, the order already shipped -> start a return
You can argue that these kinds of things should be supported via direct web UI, and that's fine, but I've seen them backed by chatbots, and I would rather do it on the web than make a phone call for a boring decision tree.
And "preferrable" meant it was preferrable to developers specifically, which based off of my experience nobody wants it there, it's an annoyance at best
Whether it saves cost or not: hiring people to manage the automated thing (including aforementioned bullshitter who is most likely expensive) or cheap front line support, I'm not sure
Some blockchain people have leveraged that into lucrative careers, albethey short.
Their actual human support couldn't figure out my problem, but instead of admitting it they just turn on AI-chatbot mode, effectively hanging up on the customer. There's no UI indication you're not talking to a human anymore. Meanwhile the AI tries to engage in small talk and tells you to patiently wait while they're fixing your problem. Eventually an actual human came back and admitted they couldn't fix the problem and was escalating me to "tier-2 support", which turned out to be another chatbot. Incredibly dystopian experience.
Meanwhile their engineers probably got a promotion for this and wrote an arxiv post bragging about their work: https://arxiv.org/html/2405.00801v2
If you make me interact unnecessarily with a chat bot, I dislike your company more, instantly. If a competitor doesn't and you are largely interchangeable, I'm leaving. The fact that Gen AI cuts down your support costs by making support worse is a net negative to me, the customer.
All we did was replace one horrible technology for another technology that does the same thing. Its not better, its not getting better and layering LLM's and AI on top of an already bad technology isn't the answer.
The only difference between chatGPT and your support bot, is lack of maturity/investments to bring their unique knowledge to have similar quality of answers as chatGPT can answer you about your queries.
The quality of the bot = money you spend * how god is the tech.
prev gen bots are still essentially a programs written by human. Build a good bot is hard , sometimes require entire team of 10+ people working every day.
Think of it on a scale :
cheap no name brand - no support
average brand - some support, almost 100% bots and documentation, super hard to get to human agent.
apple brand - ok support with bots
luxury brands - super knowledgable human been always ready to help
It's clear that we don't have resources to deliver everyone support of a luxury brand.
Luckily, we have LLM's will 100x improvements to deliver bots that are way way better.
That will allow even small brands to have a very decent support.
Chat interfaces will eventually eat western world Web 2.0, and the rest of the world is already in chat interfaces around the globe anyways.
At the absolute best, it could be an alternative to sifting through poorly written help docs.
They provide a great user interface, probably the best for creating and maintaining workflows. I still believe the prime time for chatbots is yet to come.
Those videos are created using videos and images I get from whatsapp. So to create a new video I forward those media files to my bot which is running in my computer at home and it generates the video, description, hastags, caption and send to myself (I use 2 different numbers, one for me and one for the bot). This saves me many hours of work.
This same chatbot also helps me manage my whatsapp groups.
I thought you meant more like a way of constructing or managing workflows.
We've come a long way since, but some (UX and tech) challenges remain the same.
[0] https://techcrunch.com/2016/05/29/why-do-chatbots-suck/?gucc...
The ultimate differentiator is that LLM, with proper management, can give you a definite, exact answer to your very specific question. This is also a usability boost; it's like moving from a landline phone to a mobile phone from a usability standpoint.
The entire chatbot history is a path towards making support better and more accessible because human agents cost a lot. In fact, the entire chatbot business has to compete with the cheapest human agents on the planet.
You can clearly see NLP-gen bots evolution. From a basic how to reduce the load on human agents to cover the top 100+ questions by bot.
The current LLM generation of bots will evolve into pre-AGI bots, which will be very informative and capable of answering 1000+ questions.
The improvement is 100x, at least. This is HUGE. The numbers are just average, but the limiting factor were levels of efforts to manage these bots.
People who wrote and read the article have a professional deformation/bias.
We are all advanced Google search users or even represent businesses that depend on Google. The majority of the population on earth is still not very good at doing online research to just solve their basic problems. They trust phone calls, tiktok recommendations and sometimes chatbots, sure. They will prefer anything but not complex googling and digesting large amounts of information.
LLM-powered bots are also much easier to manage, including capabilities of bots actually doing something, e.g., making requests to DB, etc.
Even pre-AGI chatbots, with their 100x+ improvement, will cut a huge amount of what we use to call web2.0/3.0.
LLM is just a vehicle that eventually standardizes content mechanisms. It will cover every aspect of content creation,update, moderation, analytics, and delivery, including, but not only delivery through a chatbot.
Such a change will be as different as the pre-internet media vs the social media era.
We used to think that the Western world is a cutting edge of every single trend globally. In fact only western world is still living in pre-chat world where websites are the main engine of information exchange.
Look at Wechat, Telegram, and Line with their billions of auditory. LLM-powered chatbots will bring the lagging Western world, which used to be golden billion, into modern world reality. This article is just a mental resistance to what's imminent.
But it's not the bots are bad.
The bot experience = $_investment * tech efficiency.
I've been in the chatbot industry for the last 6 years.
If you disable event past-gen NLP bots, millions of people will never be able to talk to a human agent or get any support at all. Unless companies invests 10x-100x more in customer care.
Even companies like Apple, in a premium segment of margins, use community forums and bots to manage their level of support.
LLM bots will bring us to a different reality when you can have 100x better customer support with the same level of investments.
And btw, that's why google is also moving this way.
I'm souring quickly on the concept
I plopped for the chatbot, then I had a problem. My apartment is wired weirdly, long story. The chatbot was very clear that it was a bot, it worked well, and it only took a few minutes of chatting before it decided to escalate to human support. The human support called me, so I didn't have to pay any phone bill, and they did it within a few minutes of bot making the decision to escalate.
All in all it was pretty satisfying. Comcast's version sounds ... less satisfying. The arXiv paper is really the cherry on top.
If chatbots really were good, and could solve my CS problems 90%+ of the time then I'd love them. But that doesn't happen, so I want to speak to a human please.
Sometimes they answer my question.
Other times they don't, and I have to wait in the support queue for a meatbag to answer.
Worth the risk of a wasted 25 seconds.
In an ideal world, any company that sells anything would be required to have a phone number that is answered by a human. The maximum hold time would be limited by law and be required to be lower, the higher the cost of the items they sell. The punishment for violation would be jail time for the CEO, at a month per violation or so.
Another way to frame this: it's a fight over whose time might be wasted.
With time being the one finite resource for everyone, stalling someone to make them go away feels unethical, when the alternative would be to just tell them to GTFO.
Technically competent customers who have exhausted all means of self-solving the problem understand this. People that call or open a customer support chat for say, a routine return to Amazon instead of clicking a button might get upset at it but certainly don't realize they're why the bots exist.
>The punishment for violation would be jail time for the CEO.
Comments on this site read more like Reddit every day. :(