Hacker News Reader

Comment by valenterry | Hacker News Reader

I'm just waiting for AI to suggest to download package X for usage in language A, which exists in language B but not A, and some malicious actor creating X in A to spread their malware.

View on HN

I keep having this happen to a very small language I used to play around with. Now when I have forgotten how it works I thought I would try to learn it again with ChatGPT and the results were pretty disappointing. It keeps using a mix of different languages, mostly C++ and Java it seems. And I hunt thrugh the documentation to find the real way to do it. It corrects itself but it's still wrong and it keeps going in circles. I guess I won't be learning that language again with help from the AI. It's just too small of a subject for the AI.

This is a demonstrative example of why an LLM is NOT intelligent at all, and we need to stop calling these models AI.

A real intelligence would be able to either put together the information needed and fill in gaps with research or it would be able to tell you that it does not know.

"Hallucination" is yet another anthropomorphism of the model. A mind hallucinates; a model just produces incorrect results.

LLMs are very cool but I'm really sick of the "intelligence" branding. If a person who didn't know about something just confidently made shit up, I wouldn't call them intelligent.

What about "Fake it till you make it" then?

I’ve just tries GPT (again, after months) on high level pseudomathematics and it blabbers on and on about nonsense…

I dont know about the newer models but you used to be able to get authoritative sounding answers asking about "the fallacy of affirming the antecedent". So you don't need to be so high level.

Well, that is really good to know. I was just reassuring myself about its nonsense generation.

Terrance Tao, who is a pretty good mathematician, made a post [1] not too long ago about how he found ChatGPT to be at least slightly useful for solving a non-trivial math problem. His basic insight is to not ask for an answer directly, but to ask auxiliary questions and for suggestions of how to approach the problem, which eventually lead him down the right path. Basically how you would discuss the problem with a colleague how didn't know the correct answer of the top of their head either.

[1] https://mathstodon.xyz/@tao/110601051375142142

Frankly this sounds very close to rubber-ducking. I'm sure LLMs throw out some creative seeds, but they're much better at coming up with creative ideas than truth. I think people are too used to getting creative ideas from other people, where the ideas are at least superficially useful, to understand that they're dealing with the world's best improvising rubber duck.

Exactly. Humans are intelligent, and we never misremember anything, and we definitely never are sure we remember a thing correctly when, in fact, we don't.

And I agree with you, I know people who make shit up about something they don't know about, and it goes "actual humans > LLMs > these people".

I think this train has passed, AI has been accepted as the name and has now changed ownership to the public. This is how live languages work and we older people are supposed to be crying "But this is not what it means!"

Well, now it does I'm affraid :-)

I know we have had this conversation with our daughter several times and we have three languages to work with...

Your correct. Its ML/LLM at heart but now accepted as to the word "cloud".

Where all the "cloud" is, is an server in a datacentre with a fancy gui.

I guess we've needed a new buzz world to keep the population excited.

Some younger guy was trying to explain that I used the word cluster wrong when talking about clustering pins on a map. Cluster was a collection of servers in a server room. He had never heard about k-means :-)

I like to think of them as Artificial Idiots.

> If a person who didn't know about something just confidently made shit up, I wouldn't call them intelligent.

It seems to be a good recipe for success in many fields (business, politics, art, apparently somewhat in social sciences if you make up the evidence for it too) so I'm not too sure about that!

The poster said “intelligent” not “successful”. I can name several “successful” politicians I would never, ever call intelligent.

I think you want "intelligent" to mean "ethical". Those politicians are definitely intelligent, they've devised a successful strategy to gain power. You could argue a successful sprinter doesn't need high intelligence, but politics is all about it.

No, seriously, one of our PMs was clearly as dumb as a metaphor for something really dumb. Rhodes scholar too, which shows you what that's worth these days.

I don't have a clear view on whether LLMs are intelligent or not, but I don't understand your argument at all. Why couldn't an intelligent agent exist that refuses to admit they don't know something, and just make stuff up? I've known people who come pretty close to this behavior.

> If a person who didn't know about something just confidently made shit up, I wouldn't call them intelligent.

Then you have a highly flawed understanding of humanity... to the point you should check out of tech for awhile and go read about psychology instead.

Extremely intelligent people confidently make shit up when they don't know the answer all the time.

Why wait? You can already try and publish malware to package registries using misspellings of names of popular packages. Of course in practice, the maintainers of the registries have a vested interest in preventing this kind of behavior and probably have multiple approaches to detecting if people are doing stuff like this.

Reply on news.ycombinator.com