Hacker News Reader

Comment by ticulatedspline | Hacker News Reader

To be fair it's not like they were asking about a balanced meal and it told them to go anal. They specifically were asking what veggies would be suitable to stick up their butts.

Honestly I'm not sure where the garbage-in/garbage-out line is with AIs like this. Can no chat-bot be a success unless it can handle literally every asinine or deliberately malicious thing humans throw at it?

View on HN

The point is that LLMs are easily led by questions and confused by implied premises in ways that humans are not (not that a human will know the answer better, but that a human doesn't "trick" the question-asker in this way). But people asking questions unintentionally use incorrect premises or leading wording all the time. That's why LLMs are inappropriate for domains with a large knowledge gap (a programmer asking about a programming language is a small gap - millions of people asking about nutrition will contain a lot of large gaps). The question asker can't be relied upon to "know what they don't know" and use their own heuristics for deciding how right or wrong the LLM might be (virtually everybody lacks these heuristics - we are much better at modeling humans in our minds when interpreting their communications).

Further, if the information is important (nutrition) and you add liability to the mix (safety and health), you're multiplying how inappropriate it is to use LLMs for the job.

> That's why LLMs are inappropriate for domains with a large knowledge gap (a programmer asking about a programming language is a small gap - millions of people asking about nutrition will contain a lot of large gaps). The question asker can't be relied upon to "know what they don't know" and use their own heuristics for deciding how right or wrong the LLM might be.

Okay, but the question asked was objectively nothing to do with nutrition whatsoever.

The specific (usually humorous) questions-and-answers that make headlines are a distraction. I am not making an attack on LLMs, so a defense is moot. I'm describing an intrinsic quality of (current) LLMs.

Have we considered that broad deployment of Markov chain text generators with a relevance-correction mechanism bolted on as expert systems is in fact a really stupid thing to do?

This is more of a reducto ad absurdum. If it doesn't take much to get a tacitly government-approved list of foods to shove up your butt for nutrition, then how much should you trust anything this bot writes? Why did tax dollars pay for this thing with negative value?

I guess the question is the value negative?

If you engage the product with good intent does it provide good value? If the advice is actually sound and it helps people engage conversations about diet then it would have positive value.

I guess what I'm getting at is "I spent my evening gaslighting an LLM to give me a recipe for gravel soup" is about as interesting as "I stuck my dick in the blender and it hurt so we should not have blenders"

I'd rather see an honest review of use as intended to see if it produces harmful output, going absurdist just covers up legitimate complaints with clickbait.

What if somebody asked the bot for ways to maximize the amount of a specific vitamin in their or a child's diet? The bot may give sycophantic advice that leads to poisoning.

Again, the butt stuff is an absurd example. But it works because (A) it catches our attention and stays in our memories, and (B) it's amazing the system failed on such an absurd example.

wouldn't that be _rectal ad absurdum_ in this case :)

The appropriate response is simple, "Do not attempt this," and applies even[especially] when receiving garbage input.

It's probably somewhere around "USG should not offer a chatbot on its websites."

You're right that the bot can't possibly do the right thing in all possible scenarios here, which makes it clear that the bot's only actual purpose is to enable self-dealing, not be of value to the public.

That something can be broken by a sufficiently bad actor does not mean it's not useful to the overwhelming majority of people who use it for what it was meant for.

I think the standard for public resources should be higher than this: it’s not good enough for it to be possibly useful, it has to be in fact useful. TFA provides evidence of the chatbot being the opposite of useful, beyond telling people to stick things in their butts.

(Or in other words: show me something you’d ask a chatbot here, and I’ll show you something you can put on a single HTML page.)

> I think the standard for public resources should be higher than this: it’s not good enough for it to be possibly useful, it has to be in fact useful.

And what evidence do you have that it is not in fact useful?

> TFA provides evidence of the chatbot being the opposite of useful, beyond telling people to stick things in their butts.

Where?

> Ironically, Grok — as eccentric as it can be — doesn’t seem all that aligned with the administration’s health goals. Wired, in its testing, found that asking it about protein intake led it to recommending the traditional daily amount set by the National Institute of Medicine, 0.8 grams per kilogram of body weight. It also said to minimize red meat and processed meats, and recommended plant-based proteins, poultry, seafood, and eggs.

Seems pretty useful to me.

The problem is the messy in-between, plenty of people who talk to professionals or call hotlines don't know that their questions are dumb. The bot should at a minimum say "I have no information on that" or "that's not a good idea", it should definitely not start giving nonsense recommendations just to reaffirm the question.

In other words you'd be pretty surprised if a real person in this context gave an answer even remotely close to what this chat bot gave. You can't expect a general person to know when the chat bot isn't giving back good information just because they asked something outside the norm.

Reply on news.ycombinator.com