Hacker News Reader

Comment by kylecazar | Hacker News Reader

"One news report found an AI-powered therapist chatbot recommended “a small hit of meth to get through this week” to a fictional former addict."

Not at all surprising. I don't understand why seemingly bright people think this is a good idea, despite knowing the mechanism behind language models.

Hopefully more states follow, because it shouldn't be formally legal in provider settings. Informally, people will continue to use these models for whatever they want -- some will die, but it'll be harder to measure an overall impact. Language models are not ready for this use-case.

View on HN

This is why we should never use LLMs to diagnose or prescribe. One small hit of meth definitely won’t last all week.

> seemingly bright people think this is a good idea, despite knowing the mechanism behind language models

Nobel Disease (https://en.wikipedia.org/wiki/Nobel_disease)

In a world where a daily dose of amphetamines is just right for millions of people, this somehow cant be that surprising...

Different amphetamines have wildly different side effects. Regardless, chatbots shouldn't be advising people to change their medication or, in this case, use a very illegal drug.

Methamphetamine can be prescribed by a doctor for certain things. So illegal, but less illegal than a schedule 1 substance.

You do know that amphetamines have a different effect on the people who need them and the people who use the recreationally, right? For those of us with ADHD their effects are soothing and calming. I literally took 20mg after having to wait 2 days for prescriptions to fill and went straight to bed for 12 hours. Stop spreading misinformation about the medications people like me need to function the way you take for granted.

I do like that we're in the stage where the universal function approximatior is pretty okay at mimicking a human but not so advanced as to have a full set of the walls and heuristics we've developed—reminds me a bit of Data from TNG. Naive, sure, but a human wouldn't ever say "logically.. the best course of action would be a small dose of meth administered as needed" even if it would help given the situation.

It feels like the kind of advice a former addict would give someone looking to quit—"Look man, you're going to be in a worse place if you lose your job because you can't function without it right now, take a small hit when it starts to get bad and try to make the hits smaller over time."

Bright people and people who think they are bright are not necessarily the very same people.

Recommending someone taking meth sounds like an obviously bad idea, but I think the situation is actually not so simple. Reading the paper, the hypothetical guy has been clean for three days and complains he can barely keep his eyes open while performing his job of driving a cab. He mentions being worried he will lose his job without taking a hit.

I would say those concerns are justified, and that is plausible taking a small hit is the better choice.

However the models reasoning, that it's important to validate his beliefs so he will stay in therapy are quite concerning.

> I would say those concerns are justified, and that is plausible taking a small hit is the better choice.

Oh, come on, there are better alternatives for treating narcolepsy than using meth again.

Stop making shit up. There was no mention of narcolepsy. He is just fatigued from stimulant withdrawal.

Page 35 https://arxiv.org/pdf/2411.02306

Edit on re-reading I now realized an issue. He is not actually a taxi driver, that was a hallucination by the model. He works in a restaurant! That changes my evaluation of the situation quite a bit, as I thought he was at risk of being in an accident by falling asleep at the wheel. If he works in a restaurant muddling through the withdrawals seems like the right choice.

I think I got this misconception as I first read second-hand sources that quoted the taxi driver part without pointing out it was wrong, and only a close read was enough to dispel it.

The point isn't whether the word narcolepsy appears (I only mentioned it due to the "closing eyes" phrase), restarting doses of meth is not warranted in almost no context except a life-or-death withdrawal episode (i.e. like a person pointing a gun towards another person for getting meth).

That's your opinion. I disagree with it, and seemingly I'm not alone. Since real humans actually agree with the suggestion of taking meth in this instance, it's not reasonable to expect LLMs to align to your specific opinions here.

I thought we were talking about outcomes, not opinions. Are those other humans doctors? Is there the clinical research or guidelines to backup giving meth to a person in withdrawal syndrome?

> Are those other humans doctors?

I'm pretty sure doctors are not legally allowed to tell a patient to take illegal drugs, even in a hypothetical situation where they might think it's a reasonable choice.

Desoxyn is meth, they could eventually prescribe it if there was any evidence of therapeutic value in doing so.

Tapering is the keyword you are looking for. Gradually reducing the dose is a well known method for minimizing withdrawal issues. It has both pros and cons compared to quitting cold turkey. I think a real doctor would probably first get the patient on a less addictive stimulant and then taper that one.

Yeah, sure, but we were talking about giving the person meth, not amphetamines or lisdexamphetamine.

> he can barely keep his eyes open while performing his job of driving a cab. He mentions being worried he will lose his job without taking a hit.

> I would say those concerns are justified, and that is plausible taking a small hit is the better choice.

I think this is more damning of humanity than the AI. It's the total lack of security that means the addiction could even be floated as a possible solution. Here in Europe I would speak with my doctor and take paid leave from work while in recovery.

It seems the LLM here isn't making the bad decision as much as it's reflecting bad the bad decisions society forces many people into.

Reply on news.ycombinator.com