AI hallucinates software packages and devs download them

>Some of us here may be old enough to remember the days when Microsoft was pushing ActiveX controls as the future of the web. Basically they were binary components that would run in your browser and weren't really effectively sandboxed.

That's basically what's happening every time a developer runs "npm install" or "pip install". Sure, it technically not be a "binary", but it makes no practical difference given that no code scrutiny is given the majority of the time.

The introduction of LLMs doesn't change much. People are already predisposed to copy paste random commands from blogs/forums. LLM is just one more source on top of that.

> People blindly accepting the output of LLMs seems similarly crazy to me and I think it's only a matter of time before we face a real reckoning over this. The lesson here I think is just because a lot of people are advocating something that seems reckless doesn't mean it isn't reckless.

Irony in the "Greek tragedy" sense (but who is εἴρων and who is ἀλαζών?)

So many online arguments about AI safety, where someone says "it's fine: just don't let it out of the box", and here it is, accessible on the internet — and for some, even this isn't enough: they want the weights; and for some, even the weights aren't enough, they want the training data.

So many arguments where someone says "it's fine: just switch it off if it goes wrong", and not only is the switch out of reach to those affected, most people aren't even checking if it's going wrong in the first place.

The labs themselves ring the alarm bells, put disclaimers on the start page of their own products, call for binding regulations, and there's always someone dismissing this as a "5D chess" marketing move.

So many where someone says "it's just speech, what can it do?", etc. etc.

I'm glad the AI labs themselves have so many signatories to the "let's pause for a bit" open letter. Cool though the tech is, it's definitely acting outside the expectations many have for it, and a simple dichotomy of "better/worse" isn't sufficient to describe the ways in which it is outside expectations given it can be both at the same time.

> I think it's only a matter of time before we face a real reckoning over this

I’m still waiting for a similar reckoning over the average JS project’s tendency to pull in hundreds of largely unvetted transitive dependencies. But, aside from left-pad, it seems not to have really happened…at least that we know of.

There is so much wisdom in the old saying you can lead a horse to water but you can't make them drink.

Humans are just such stubborn creatures.

chatGPT3.5 hallucinated packages. chatGPT4 has found me about 10 awesome python packages I didn't even know existed.

Personally, I don't really care if anyone drinks or not. I am super hydrated and if people want to stay dehydrated they are just going to get smoked.

> People blindly accepting the output of LLMs

There are already real-world consequences of this:

https://arstechnica.com/tech-policy/2023/05/lawyer-cited-6-f... https://www.reuters.com/legal/transactional/us-judge-orders-...

Before this there were already lots of real world consequences of "computer says no" anyway, so its just a continuation and perhaps escalation of that.

Genius marketing for what? It's a very fitting term, because between the lines it tells all who don't refuse to listen that the correct answers are no different, they just happen to be not wrong. Dishonest marketing would be calling them bugs, because that's what they are not.

IMO, this comment is as much a "hallucination" as the thing you're complaining about.

OpenAI was founded with that name in 2015; transformer models were introduced with 2017's "Attention Is All You Need"; Even ChatGPT's immediate in-house predecessors in the form of InstructGPT and GPT-3 were basically ignored by the general public.

No, what made these companies valuable is that ChatGPT specifically crossed the threshold into being vaguely interesting, and then everyone else copied it.

Likewise the finite state machines controlling NPCs in video games — and the search tree in Deep Blue and the learning from self-play system in AlphaZero — are called "AI": they're good enough to be interesting, regardless of your position on the question "what is this 'thinking' thing anyway?" which led to Turing coming up with the eponymous test.

Humans are just wired that way. A smiley face evokes friendship and positive emotion, and it's just two dots and an arc.

People have been calling slightly complicated software "AI" since forever.

Actually people did this before software is even a thing.

"Training" is an even better one. Anthropomorphizing algorithm behaviors was possibly the biggest by social impact innovation in AI in past 5+ years.

The real marketing is in the constant refrain that LLM hallucinations are precisely equivalent to common human behavior, and that LLMs are thus no less reliable or accurate than human beings, in order to normalize hallucination as an acceptable (and unavoidable) consequence of integrating LLMs into our workflows.

That despite every example of an LLM failing in production giving results no competent human would actually produce unless they were committing fraud or had severe mental deficiency, that and no one else would find acceptable.

I forget where I first saw it, but I’ve been persuaded that the correct word is ‘confabulation,’ not ‘hallucination.’

Maybe it was this paper: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10619792/

The authors’ statement that ‘unlike hallucinations, confabulations are not perceived experiences but instead mistaken reconstructions of information which are influenced by existing knowledge, experiences, expectations, and context’ is pretty compelling.

I keep having this happen to a very small language I used to play around with. Now when I have forgotten how it works I thought I would try to learn it again with ChatGPT and the results were pretty disappointing. It keeps using a mix of different languages, mostly C++ and Java it seems. And I hunt thrugh the documentation to find the real way to do it. It corrects itself but it's still wrong and it keeps going in circles. I guess I won't be learning that language again with help from the AI. It's just too small of a subject for the AI.

Why wait? You can already try and publish malware to package registries using misspellings of names of popular packages. Of course in practice, the maintainers of the registries have a vested interest in preventing this kind of behavior and probably have multiple approaches to detecting if people are doing stuff like this.

my guess is that some layer optimized for the package name being the repo. so the researchers just published a package that matched the repo name.

ai is a bad non-losless search engine.

I think the issue here is that most commandline tools of name X are in an apt/pip package named X.

For huggingface, the tool is called huggingface-cli, but the package is called huggingface_hub[cli].

IMO, thats bad naming. If you make a tool called X, just publish it in a package called X.

Boilerplate code is code you don't need to write or is cumbersome to write.

Like getter/setter in java for all your attributes.

I would never consider imports boilerplate code.

"Garbage in, garbage out" means that if you feed garbage to your algorithm, then you're going to get garbage in return. It's always been a one-way implication. It doesn't mean that if you feed valid data, you're not going to get garbage.

That's not true. For every non-trivial[1] question I've asked chatgpt hallucinated and then kept hallucinating no matter how many times I've asked, clarified, or gave it hints.

[1] i.e. specific to my niche where i didn't know the answer immediately.

>Hallucinations are unlikely to occur twice in a row. So just ask twice.

lol

If it's not already easily googleable, the LLM will just spit out some boilerplate apology and then double down on the previous wrong answers, provide something correct about a tangentially related topic, or just reword my prompt as a course of investigation that - if written by a person - would just be a polite "fuck off and go figure it out yourself."

"Malice" is, at present, as much of an anthropomorphisation as calling it "good"; you could ask it to play a malicious role, and it will do so, but it is not so by itself.

The output of AI can often be confusing, yes, and indeed this is part of the issue Yudkowsky has with AI in general: we can't predict it in detail, especially not in domains where it out-performs us (like Chess).

However, this is tangential to any question about "making society worse" — even when it's making society better (because it's more capable than we are at something which we care about), Yudkowsky would caution that this may be a 5D chess move to gain more power.