Kinney Drugs pulls back AI phone assistant after hundreds of customer complaints
wcax.com
wcax.com
As a consumer, the errors that AIs make are more than an annoyance. I have yet to meet an AI phone assistant that can do anything more than an explicitly programmed phone tree, and several, including the one for the pharmacy I use, that do far less. They are undoubtedly much more expensive to create and test.
Clearly there's a bubble going on, and unprincipled people chasing funding and promotions as usual, but why are companies like CVS paying more to get less?
I think it is literally to waste our time. It's one more layer of defense to stop you from talking to a person.
There are two big wins in health care. One is to increase the scope of health care that gets paid for. More and more expensive treatments, to bring the money in. The second big win is to stop people from getting treatment. Deny coverage, confuse them and wear them out with bureaucracy, and in this case stop them from talking to an expensively educated human pharmacist. That's what an AI does better than a phone tree, because it more effectively creates a sense of helplessness and valuelessness in the customer.
Expanding the scope of care you're entitled to and then fighting against your ability to get it isn't a contradiction any more than it's a contradiction when a person breathes in and breathes out. It's two synergistic parts of a system whose goal ultimately has nothing to do with health care.
LLMs perform pretty poorly when you start try to restrict them too much (e.g. detect intents and control responses based on intent detection), the replies gets disjointed and weird feeling. An open ended prompt and chat completion gives a much better feeling most of the time but can go off the rails.
If you don't let LLMs handle the conversation entirely you get into insanely complex traditional engineering problem: (if user is complaining do x, if user asks a relevant question do y, if user asks an irrelevant question do z), which is an endless can of worms.
So yes, death does happen by phone systems.
This was in 2013 pre AI.
See our documentary Pain Warriors about that whole saga. Free to watch on TubiTV and Amazon Prime. I get no renumerations of any kind. The opposite actually, has cost me greatly. Pain Warriors has won several awards.
AI unfortunately is not deterministic. It would be nice if you could actually use it deterministically in this way, set a seed such that the logic flow are preserved across all sessions well into the future unless the logic flow instructions are changed. They just don't work like that though. Even if you have a good harness they can go off the rails pretty easily and need coaxing to get back to appropriate context. That is a nonstarter for the typical caller of a call center. Functionally speaking just an automated call where you touch type through options would be equivocal to the human operator and that paradigm has replaced a lot of call center work already, with the human operator serving to smooth things over if the person is old or can't comprehend the options for whatever reason and starts yelling incoherently into the automated line (my actual strategy for getting human operators on the line).
Everything becomes a nail though when you have a hammer though, even more so when you are selling hammers. So expect to see AI in plenty of places where it makes no sense to belong given the supremacy of existing tooling. There is also no advocate for a lot of existing tooling unlike AI that has sales teams active in pursuit of new customers integrating this into their product.
With something like healthcare though I'll remain eternally skeptical that anything other than a licensed expert will be palatable to the public.
First, The technology works, and it scales, but the whole bottleneck is domain expertise and implementation. These are expensive, and hard to scale. We hire pharmacists as project managers, that's how important domain expertise and implementations are.
Second, the amount of noise of "Voice AI for <industry>" is incredible. Most of them are completely clueless about the industry & basically "YC-striver" type who can only sell to other yc companies. They fail hard the moment they touch critical functions of the real world.
We have hundreds of pharmacies with us now and they mostly successful and happy. Our biggest friction is customers being burned by vendors like this and writing off AI altogether as "not ready".
This is such a great graph. Your company feels successful because your customer is happy yet the rest of the world would be just happy if your company died in a fire. This seems to also be just fine for companies like yours.
How happy are their patients?
People have no ideas have pharmacies work. Pharmacies don't control how much they pay for drug, or how much they get paid. The margins are terrible and you can't hire enough techs to answer the phone at peak call volume. So - most people experience are terrible as they wait on hold for a really long time.
EVEN if the AI is a little rough around the edges with relational calls, just solving for transactional calls with no hold is a massive win.
It’s nice that this remains an option. It’s a dying one. So much is better handled and more easily handled in person than over the phone/mail/web/fax/email/blah, even if they’re 1:1 with a human.
It’s why I stick with my credit union instead of moving everything over to an online bank even tho I’ve managed to get 99% of everything done with the CU without going to a branch.
Just as you said, I spent 20 minutes YELLING at the AI before it would transfer me to a human. Oh, and the entire time it was spewing ads for their programming. When I got to the human she, once again, confirmed it WAS a feature and it re-activated during 'free weekends'.
My father had cancer. After he was diagnosed he retired. His full time job became managing his health: medicare, state benefits, doctors appointments, medications, talking with billing people. He was on firstname basis with multiple providers. He invited some nurses and their families to the first 4th of july when he was cancer-free (they politely declined to keep the relationship professional).
I just imagine my father, or my 75yr old mother (whose health is beginning to decline) trying to navigate this new AI world. It must be an absolute nightmare. I can imagine how frustrated and disempowered they'd feel after a lifetime of reasoning with other humans.
So "AI" treats relatonal as excluding transactional??
You keep saying that reducing call time is the "win." It seems like that the only metric you're measuring. If so, your product is a failure, not a success.
This is HEALTHCARE. You don't get to do "rough around the edges." You don't get to dismiss edge cases.
I also work in healthcare tech. My company doesn't permit such things because people die. I'll write it again because it's a point that tech fetishes can't get into their heads:
People die.
Good for you - I want in the ICU delivering care during COVID as a frontline clinician. You don't get to pull this line.
I live in a major metro and there are no pharmacies left except the largest ones. All my problems have to do with insurance not covering things, and insurance asks me to ask the pharmacy, pharmacy asks me to ask the insurance, and often insurance then asks me to ask my doctor to adjust the dosage cause they won’t cover that many days. Customers are batted around and I don’t see any AI that will solve this problem which is structural. The only value I see in AI is cost savings to replace humans, which is to say practically none as a customer over a website to check status and try to refill things.
If that’s how you’re measuring patient satisfaction, you’re completely clueless.
But I thought the issue with customer surveys was how biased they could be toward negative experiences cuz nobody else is angry enough to respond.
They can be biased toward positive experiences too, if customers feel the job of the person that helped them is on the line.
NPS is asking your customers "how likely are you recommend us to a friend or colleague?". That's definitely not an accounting number. If you care about this, you can instead ask your _new_ customers if someone recommended you. Or you could hire a third party to do an independent survey among your target demographic.
"AI" reduces call time here... as I hang up and look for an alternative supplier.
However, when the mistake is made by the replacement that cost one of your community members their job for a few pennies gain...
The bar for these systems is higher than the bar for humans.
God I would love it if this comment is read out at a hearing about defrauding investors or massively leaking PII
Is your pharmacy one of your customers? Do you use your own product regularly?
Classical voice to text combined with NN based voice to text, I would imagine can be highly accurate, and that’s probabilistic.
Of course, you're right. I didn't intend to criticise him.
I just think that it's not a good idea to transfer this megatrend to healthcare.
Yes, those systems can be highly accurate, but as we all have experienced, this is not a stable or consistent property. A very good model can give you an ingenious answer one minute and an utterly dumb one the next. My guess is that it's simply a consequence of the extremely high complexity of the 'plant' (natural language, language interfacing with the real world), leading to some highly non linear behaviour.
I've seen AI used well in a few support situations, and it is great - faster to get to the answer vs a human (usually faster than even getting to a human, which is another issue)
When my case can be handled by the automated system, it is great, when it is not, it is usually misery, leading to anger at the customer and whatever company sold them that automated isht.
Especially for intelligent knowledgeable users of anything, by the time we call, it is an issue likely to need escalation.
So, the one absolutely critical factor for me and everyone I know is: how fast and easy is it to get to a human when we find the automated assistance does not work for our particular issue? It should be "I need to speak to a human now", and the immediate response should be "OK, let me get you one... Hello, this is [person], how can I help?" with the prior transcript already on the human's screen.
If this works, it is good because the humans are operating a level up, not having to deal with endless monotonous minutiae, and dealing with interesting issues all day, which means they get good at it and have a better attitude — a win for all.
How, and how well do you handle that situation — how close are you to the proper response I described?
I suspect that applying an aerospace approach of dissimilar redundancy and root cause analysis to this will yield a much better experience for everyone (i.e. you, the pharmacies and the patients) than what would have been possible with humans alone.
I've ranted about this before, but medicine as it stands isn't a serious field. And I can say that with a straight face, because medicine has until now, been the only field of modern scientific endeavor (or rather cloaked under modern scientific endeavor) that has fought tooth and nail against gathering more data points for improving understanding.
More detail here, https://news.ycombinator.com/item?id=35762650
I'm glad that you're doing this!
And if not, why not?
Meanwhile you are royally fucking over the end user with your AI shit.
“You are absolutely right! You should indeed not give your 5 year old child 1000 milligrams of ibuprofen per hour”
In many ways this is a repeat of the India call center train wrecks of the 00s. On paper, letting someone in Bangalore vs onshore handle incoming customer service calls looked like a path to amazing savings. In practice the customer experience was horrendous and companies CTRL-Zed these decisions and rapidly brought customer service back onshore again. AI is just that story of shortsighted decisions by weak leadership playing out all over again.
I didn't mean to imply that all companies that went hard into AI will die - just that a lot of companies that follow the trend may not realize the true maintenance costs and of those that did misguidedly follow the trend I'm sure some will luck or manage their way out of disaster for various reasons.
Not here in UK. They rerouted to the Phillipines, Egypt etc.
No they didn’t, I can’t remember the last time I talked to an on shore rep.
Our user base skews elderly. Going in, I was really skeptical about how much engagement and value we could provide to these older patients. I'm blown away how well it all turned out and how valuable it's been for our patients and client clinics.
For example, we were able to reduce no-shows for some high-cost appointments by 80%. Turns out that the reasons why people don't show up are a complicated and diffuse — everything form anxiety, to not speaking english, to being confused about how to get into the building — and properly configured LLM-based systems are really, really good at dealing with exactly this sort of problem.
Like they don’t even say the full name of the prescription they just tell you the first three letters and they’ll say “for the patient born on…”
Current pharmacy IT systems are currently almost 100% automated so how are you claiming it’s the first use?
Even worse is that the human is usually just as worthless because the problem is often upstream at the insurance provider or regulation
This is something I’m finding with people complaining about AI systems the AI system in this case is not any worse than the previous IVR system was it just expanded the use cases and now people say oh it doesn’t work for those expanded use cases
There’s no mention whatsoever of the current absolute devastatingly bad state of all software across the world
It’s like people are mad when there’s an application that already sucks, someone attempts to use genAI, that fails and then they blame AI
The whole thing is absurd
https://arstechnica.com/tech-policy/2024/02/air-canada-must-...
> Air Canada essentially argued that “the chatbot is a separate legal entity that is responsible for its own actions”
Yeah, if I'm calling or chatting with a company, I expect them to stand by whatever commitments they make, and I will judge them harshly if they don't. This is true whether an AI or a human is speaking on the company's behalf.
Yes, it still screwed up. And yes, Air Canada tried to weasel out of it, but even though it was a chatbot it had no LLM behind it.
AI systems were in use for decades before LLMs rolled around.
But this doesn't seem likely to translate well to pharmacy, where there will be consequences for treating customers like vermin, as opposed to actual incentives to do it.
I wouldn't be shocked if the tech needed in places like pharmacy or other sectors where there's some level forced responsibility leads to the version that's actually palatable elsewhere.
Set up whatever online portal or app you want. I’d prefer to just sign in and click a button.
If I’m picking up the phone or sending an email it’s because I have a weird problem and I need to talk to a person.
The biggest problem with AI customer service is that a human employee would've let a minor issue slide without escalating it. But a chatbot often inflames the situation, and by the time the customer reaches a human agent, they're already furious.
Most people aren't rational or logical. Non verbal feedback, like acknowledging someone's anger and showing empathy, is incredibly important.
Then the poor wretched line workers have to handle people for whom the companies are screwing over - there is no effective line upwards to anyone making the decisions to use these things other than an impotent “feedback survey”. You can’t “ask for AI’s manager”. This accountability sink is not only horrible customer service, when it’s life and death like “can I get the perscription that allows me to live” wild stuff starts to happen.
At the time, they had a real person handling their support. The real person told me this was just a glitch and that my correct address was on file and everything is fine.
That wasn't actually true. A new debit card to replace my expiring one ended up being sent to the old address a couple of months ago.
So I go back to the website to try to edit it one last time. Same problem.
I call them and get their fucking stupid AI slopbot.
Still, I explain the whole situation:
> I'm trying to update my address on the website. When I click the edit link next to the old address, it shows my new address instead. I need to remove the old address.
What did the slopbot decide to say? Oh well, I should go on the website and edit my address of course! I gave it two more chances by trying to explain it a different way and how this didn't work.
Did it accept that and escalate the issue to a human? No. It kept telling me to go to the website.
I had to say
> Hey you fucking bitch, how many times do I have to tell you that doesn't fucking work? Escalate this to a fucking human before I get the attorney general involved.
to finally get a live person to fix it (I actually had to do this twice, the first set of fucks, the slopbot forgot what problem I was having and needed a refresher!). Suffice it to say, I will never trust any large amount of money with this bank again. I'm highly skeptical of any company using AI CS now, because this is one of my many experiences where the slopbot is incapable of recognizing a complex situation outside of its capabilities.
I mean look at this situation: their AI slopbot turned this one mistake into 5 mistakes instead of just allowing it to be one that another human corrects.
It's gotten to the point where whenever I see someone lauding AI customer support, I just assume they have an investment/job that hinges on people liking it and/or their only experiences are using it as a glorified refund button.
Oh, I’ve had this before. Apparently there was some background “registered address” and I had only updated my “mailing address”. They happily sent monthly statement to the mailing address for years but sent important notices about shutting down their entire small biz account service to my out-of-date “registered address”.
Fun times walking into a branch because you haven’t gotten a statement for a few months and they say “oh, your account has been closed, we sent a $$$ bank draft to $oldaddress”
I'm taking that information and building a voice version. I'm taking a lot of time to make sure the phone version works naturally and that is can address most common things people call in about. Just like my chat version, GPT 4o was barely good enough to do what I needed. Since GPT 5 and on it has been more than adequate. I expect the same thing over time with voice, Open AI is going to release the GPT-live to the API which is a noticeable improvement.
Some people are just offended at the idea of talking to an AI chatbot voice or phone, no matter how good it is. I don't know how good this Kinney Drugs bot was, but it's very easy to make something that works but ends up being very frustrating for customers.
You don’t need a cookie banner for functional cookies, and opt out should be a single button.
Still, I got it to send a chocolate chip cookies recipe to us.
These things are horrible, but then coding agents were dumb as dogshite not too long ago, so maybe in a few years AI support agents will become useful.
Marraffa should hire some better PR people, closed-source AI which doesn't generate or manipulate data is just a hilarious thing to say adressing data-privacy concerns;)
It's usually the big old companies who fail to realize this early on.
I think we need to take a good hard look at what we want out of this world and ask ourselves. Do we really need to automate this? Like really. What benefit does the world receive having an AI chatbot tell grandma the wrong doage for her meds? This is the naked extraction of value from a service that's already been offshored. Companies like these are gleeful to make their service as shitty as they can because they know their users are doing so against their will. Infact, that's become the goal.
it seems very implementation dependant, I know a startup with incredibly realistic voice agents that most people don’t detect is an ai that outperforms humans. But ai for CX seems like an easy project to do for companies looking to cut costs so the average agent a user encounters in the wild is poor quality
BS. How is the AI touching that information?
Prescription notification is outbound communication. It’s almost a recorded message. A little text to speech if it is supposed to read the drug name.
Wrong dosages?? This must be handled by the “enroll a new scrip” functionality, right? So basically B2B doctor interaction. You let AI touch that? Aren’t the vast majority of the costs from customers not doctors? None of this makes sense.
But the biggest problem is ASR. WER is still atrocious even with SOTA models. When you add drug names and regional accents, it's a recipe for disaster.
WER = Word Error Rate
Too many acronyms; not everybody is in your field.
And the patients generally don't have a pronunciation key. Doctors prescribing might not, either.
It ends up being quite a lot of variation.
As some examples: Wegovy, Ixempra, Qvar, Keflex
Iunno, pretty sure a car manufacturer will sue me for trademark infringement if I try to manufacture a Beetle under my brand name.
The bigger answer is regulations in some places against the drug name inferring what it treats.
If you want to do global marketing; you gotta go for lowest common denominator: something meaningless in every language/slang.
We need to fight the talking point that open source is bad!
AI couldn't do this job with a checklist. It would forget the entire checklist after two responses, then get fixated on the words that the customer used to complain about how dumb the AI was and loop until they hung up and called back to hopefully clear the context.
edit: Unless Kinney had been sold a product that used caller ID to carry over context, then the customer's best chances would be to scream "Agent!" over and over again (to get put on hold for 45 minutes, then transferred to the most screamed-at Philippine minimum-wage gig worker on the planet), or to show up to the drugstore and start screaming at people.