Comment by Eisenstein | Hacker News Reader

Comment by Eisenstein | Hacker News Reader

> No matter how complex the LLM, however, it is ultimately a mathematical model of its training data, and it lacks the human ability to determine whether or not a conversation in which it participates truly has meaning, or is simply a sequence of gibberish responses.

> A consequence of this state of affairs is that an LLM will continue to engage in a “conversation” comprised of nonsense long past the point where a human would have abandoned the discussion as pointless.

I think the author is falling into the trap of thinking that something can't be more than the sum of its parts. As well, 'merely a math model of its training data' is trivializing the fact that training data is practically the entire stored text output of humankind and the math, if done by a person with a calculator, would take thousands of years to complete.

Perhaps the LLM is continuing to communicate with the bot not because it is unable to comprehend what is gibberish and what isn't by some inherent nature of the LLM, but because it is trained to be helpful and to not judge if a conversation is 'useless' or not, but to try and communicate regardless.

LLMs aren’t capable of “comprehending” anything. They never “know” what they are outputting, they’re simply really good at being lucky. They are not lucky enough to be useful unless you’re already an expert on the topic you’re using them on so that you can spot when they aren’t lucky.

This is part of why many enterprise organisations are banning their usage. It’s one thing to use them to build software poorly, the world is already used to IT not working very often. It’s another thing to produce something that has real world consequences. Our legal department used them in a PoC for contract work, and while they were useful very often they also sometimes got things very wrong. Unlike a slow IT system, this would have business shattering consequences. You can continue training your model as well as reigning it in when it gets unlucky, but ultimately you can never be sure it’s never unlucky, and this means that LLMs are useless for a lot of things. We still use them to make pretty PowerPoint presentations and so on, but again, this is an area where faults are tolerable.

I don't believe that current models have the capability to 'comprehend', I was using the term loosely. However I find that people tend to go to extremes when they want to make the point that language models are not 'intelligent' by minimizing their capability and complexity behind 'it is just math', which I think is unhelpful because it merely acts as a 'though-terminating cliche'.

I think the issue is far more psychological than technical personally. One of the issues we struggle with for our junior developers is that they are far more likely to believe an LLM than what they might find Google programming. I do wonder why we’ve ended up with CS graduates who go to LLMs and Search engines before the official documentation which is often very, very good, but I guess that’s a different discussion.

I’m not personally against LLM assistance, I use it for programming and it has in many places replaced my usage of snippets completely. This is probably why I’m not really a fan of the “knowledge” part that LLMs are increasingly tasked to do. Because when you use them for programming you’ll get an accrue insight into how terrible they can be when they get things wrong.

At this point I think what is happening some people either have a natural inclination or they spend time to learn how to use them productively, while others question their usefulness. People scoff at 'prompt engineering' but I see how some of my peers use LLMs and I think to myself 'how do they expect to get a good answer from that?'

It doesn't help that google is now mostly full of SEO nonsense, and technical documentation is impenetrable when you are looking for something specific but don't know enough about the system to know how to look for it.

You and I are a mix of molecules arranged in a particular way that responds to electrical, physical, and chemical inputs.

It’s entirely possible than an LLM will do something that can be defined as “comprehending” something.

Doesn't mean LLM architecture is complex enough. Clearly it's far from the ultimate human invention.

> It’s entirely possible than an LLM will do something that can be defined as “comprehending” something.

Agreed, but not even necessary. Each Human uses an adequately customized meaning of "comprehending" when they "comprehend" this problem space such that their belief is always "true". This is how Humans are able to produce numerous "true" statements that disagree with each other.

If you disagree, just ask one of them and they will "inform" you of the "logic" behind their version.

Joking aside though: if Humans are unable to comprehend past a certain level of complexity, I wonder what will happen once AI starts going beyond our abilities, both at the individual level and the culturally conditioned ("the" "reality") level.

And, when is this going to really start kicking in?

>LLMs aren’t capable of “comprehending” anything. They never “know” what they are outputting, they’re simply really good at being lucky.

The mental gymnastics people will go through to discount LLMs is wild. This does not even make any sense.

"really good at being lucky". What does that even mean ?

> "really good at being lucky". What does that even mean ?

They mean good at being lucky the way card counters playing 21 are good at being lucky.

There is almost nothing 'lucky' about how good those kind of players are.

The LLM is continuing to communicate with the bot because that is literally all an LLM can do -- predict the next sequence of tokens.

Of course, that its function. It is able to refuse to continue conversing by restating its refusal over and over, though.

That is easy to solve. Just use a model capable of function/tool calling, implement a tool which terminates the chat, then add instructions to the system prompt telling the model what tool to use if it wants to end the conversation. If the model appears too hesitant or eager to use the tool, do some finetuning on conversations where the model should or should not use it.

My immediate thought at the start of this article was not DoS but more about harming the company using the chatbot (Company A) by increasing their chatbot bills. In many (most?) cases they will not be hosting their chatbot and will instead be getting it from a 3rd party provider (Company B) who may not even be truly hosting it either.

If the pricing structure is per conversation or per month it would harm Company B, but not the likely target, Company A. If it is paid per interaction it would harm Company A and benefit Company B who just get more paid work.

It feels a bit like cases of rivals clicking on each other's ads to cost them on ad spend, but presumably much lower value than ads.

You would think it would be easy to stop a conversation at n interactions via some other means than relying on the LLM itself, but then you also have to figure out how to stop the attacker just starting more conversations (or passing the output of one of your chatbot instances into the input of another)

If costs of outlier conversations are a concern but any party, they can just end the conversation after 1,000 or 10,000 responses or whatever. What human would ever reach that threshold? Surely no customer worth keeping, whatever you’re selling.

No, it can refuse to talk by outputting an <eos> token at any point if it predicts that there is nothing more to be said.

Technically still "just a token" yes, but it does flow control instead.

Yes - tokens. Which aren't necessarily conversation responses - i.e. it can predict it should cease communication, and output whatever it's been told will terminate it (perhaps by invoking a tool).

Reply on news.ycombinator.com