Hacker News Reader

Comment by fzeindl | Hacker News Reader

> Prompt injection works because LLMs are dumber than humans at keeping secrets, and humans can be coerced into revealing.

I wouldn't say dumber than humans. Actually prompt injections remind me a lot of how you can trick little children into giving up secrets. They are too easily distracted, their thought-structures are free floating and not as fortified as adults.

LLMs show childlike intelligence in this regard while being more adult in others.

View on HN

The amount of anthropomorphizing of these LLMs in this thread is off the charts. These language models do not have human intelligence, nor do they approximate it, though they do an incredible job at mimicking what the result of intelligence looks like. They are susceptible to prompt injection precisely because of this, and it is why I don't know if it can ever be 100% solved with these models.

"It merely has all of the byproducts of intelligence, its not intelligence though!"

I make this statement in a frank way to rhetorically get the point across. I find myself continually surprised by the general community's desire to reject the intelligence claim in its entirely. I make no claim that this intelligence manifest in the same way human intelligence does. I make no claim that this intelligence can even be measured in the same way a humans intelligence does. What I do claim though is that it is intelligence - intelligence that relates to humans in the same way the mind of a crow might.

The dominant mindset I have observed in my life thus-far when people discuss human intelligence is the pattern matching perspective. Humans are differentiates by our outsized ability to pattern match being able to successfully manipulate these patterns. We now see something nonorganic with amazing pattern matching abilities. We have previously seen other organic entities with impressive pattern matching abilities. Why must this situation be any different?

My overall claims:

- Intelligence is best measured by outcomes. How some entity is best able to manipulate its existence (however that existence may manifest)

- Intelligence can manifest in more than one way. An entirely mechanical system could be considered to have some level of "intelligence"

- Considering something intelligent or to have desires is not anthropomorphizing. There are many non-human entities that we consider to have these properties.

> Intelligence is best measured by outcomes. How some entity is best able to manipulate its existence.

I agree, although I think we humans have always been fairly bad at measuring intelligence in a way that truly appreciates all the complexity of it. The second part of that is also interesting and I would agree that is partly what makes these LLMs non-intelligent. The models do not really have "an existence" outside of the moment in which they are processing the context and producing output.

> Intelligence can manifest in more than one way. An entirely mechanical system could be considered to have some level of "intelligence"

I don't think I agree with this, or at least maybe I disagree with your definition of "intelligent". I believe that intelligence is heavily intertwined with biology and it exists is all manner of non-human creatures but I don't think I would call an entirely mechanical system "intelligent". Perhaps I would say it had "intelligent design".

> Considering something intelligent or to have desires is not anthropomorphizing.

I absolutely agree with this and I was not trying to imply that it was unique to humans. In fact I think we severely discount the amount of intelligence in non-human life forms all the time.

I do think that ChatGPT possesses knowledge (as encoded in its weights) similar to a book, however unlike a book it also has a convenient and familiar interface that allows us to interact with this knowledge and form unique and novel results.

You can define “intelligence” to only refer to biological intelligence. But that doesn’t mean that AIs can’t do things we call intelligent in humans at or beyond a human level.

Category error. You should look up the definition of intelligence. Your definition is unorthodox.

Intelligence is a much more encompassing term that describes abstract reasoning ability among others.

All biological systems are intelligent to varying degrees, but not all intelligent systems are biological.

Because while it’s mimicking a human kind of intelligence, it’s missing the kinds of intelligence that even basic mammals have.

One example: it has no concept of objects and permanence. Something even my dog has.

Want an example? Watch Gotham Chess on YouTube play it at chess and you’ll see it not only doesn’t understand the rules of the game, it can’t even remember which pieces are on the board!

Dogs can't use complex language at all, so it's also missing a kind of intelligence that the model has. It is not surprising that a pattern matching device tuned to just text (and some 2d images I believe?) doesn't have a great understanding of concepts that are obvious in the physical world. It's more surprising that it is often able to approximate pretty well without having any first hand data about it.

You’re right, dogs can’t do that. That’s because intelligence is clearly a multi faceted and extremely complex concept to define. It’s so difficult to define, in fact, that it seems we’re only able to do so by pointing at things and going “that’s not it.”

That’s not moving the goalposts at all. If it were, we’d have stopped at search algorithms back in the 60s and declared AI to be “solved.”

You mention the physical world, so let’s talk about self driving cars, the last thing we thought would be “AI” just a decade ago. It’s 2023, and Tesla still can’t stop their cars driving into concrete barriers. Something that the system was built for and, again, pretty much every animal can do without thinking.

All of these systems and research definitely get us closer to understanding intelligence (and maybe creating AI one day) but to say they are intelligent is to ignore your own intelligence that knows they obviously are not.

This type of comment sounds like the one that comes up anytime someone mentions “serverless”.

“Well there is no such thing as serverless. There are servers in the background”.

Yes people on HN already know that. We also know that Alice and Bob are not real people working in cryptography.

Yeah no I don't believe this is a fair comparison at all, and I'm frankly surprised you think this is accurate to the discussion around LLMs. There are certainly people on here who believe and talk about ChatGPT as if it is generally intelligent. I suppose if you really want I can look through previous threads, but you really can find this under most threads about ChatGPT. A brand of this fallacious reasoning I find particularly annoying are responses that take the form of "well humans also do <reductive vague parallel to LLM operation>" usually in response to people pointing out weaknesses in these language models. It doesn't really matter whether these commenters believe it or not, it does not further the discussion in a meaningful way and it perpetuates FUD around the "AI takeover".

on a technical level, can you explain the difference between pre-trained transformers and human language processing? why does this difference make them more susceptible to prompt injections than we are to—say—lies?

I’m not saying you’re wrong, I just want to see your working

I think "childlike" comes close but misses the mark a bit. It's not that the LLMs are necessarily unintelligent or inexperienced - they're just too trusting, by design. Is there work on hardening LLMs against bad actors during the training process?

Yes there is. The paper describing GPT4 had a section on this.

How about: unreliable child savant.

You can use it in sentences such as:

Would you let your children talk to the unreliable child savant?

Reply on news.ycombinator.com