Hacker News Reader

Comment by ilaksh | Hacker News Reader

The user enabled a GitHub ChatGPT plugin and authenticated with GitHub, then was surprised and annoyed when, after he complained about an issue with a project, GPT-4 created an issue for him, using one of the commands provided by the plugin.

PEBCAK.

View on HN

It's still surprising when you see it do something like this for the first time.

I wrote a plugin to give ChatGPT access to execute plugins in a Docker container [0]. The first time it said something like "I'm going to use Python for this, oh, it's not installed, I'll install it now and run the script I just made", I was pretty amazed.

What I've come to realise is that although ChatGPT is excellent at telling _people_ how to interact with systems, it's not very good at interacting with them itself, as it isn't trained to understand it's own limitations. For example it knows people can run dmesg and look at the last few lines to debug some system problems. But if ChatGPT ran dmesg, the output would blow through the context window length and it'd get confused.

[0] https://github.com/dave1010/pandora

It also has been largely trained on people talking about doing things and not actually on doing those things.

I wonder if that kind of training data is being worked on now..

I guess we'll need to train ChatGPT on millions of lines of bash history. Maybe some sort of augmented one that is pieced together with other context.

Compute scales quadratically with increasing context window sizes. It's not reasonable to achieve this with current network architectures.

The plugin is supposed to ask for confirmation, according to OpenAI's documentation at least.

> When a user asks a relevant question, the model may choose to invoke an API call from your plugin if it seems relevant; for POST requests, we require that developers build a user confirmation flow to avoid destruction actions.

https://platform.openai.com/docs/plugins/introduction

That's why he had to authenticate with GitHub before it could do anything on his behalf.

That doesn't satisfy the requirement for a confirmation on each post request though

It does because the requirement is only for destructive actions

The requirement is for POSTs in general, not just deletes. Anything that does actions outside of the system instead of just getting data

The wording is ambiguous, but probably should apply. Also the rule should not be based on HTTP verbs at all since that's the wild west.

What requirement is that the specific text is “for POST requests, we require that developers build a user confirmation flow to avoid destruction actions.”

a) it says the require the plug-in developer so, not the ai

b) it’s scoped to destructive actions which is a subset of post requests

This was added recently, and is not enforced as far as I can tell.

See my blog post: https://embracethered.com/blog/posts/2023/chatgpt-plugin-vul...

That's terrifying. This simple requirement would be trivial to enforce automatically, and yet nobody gives a fuck.

It's unbelievable how fast-and-loose people are playing the topic of AI safety. If a strong AI is ever actually developed, there is no chance it will be successfully contained.

That's victim blaming. They didn't overtly complain about an issue with the project, they were asking for usage guidance. This is the kind of contextual inference which language models are supposed to be (relatively) good at. The whole sales pitch of GPT-4 is a do-what-I-mean interface, and it clearly did not do what they meant, or what any reasonable hacker would expect them to mean.

> that's victim blaming

No, it's not. Victim blaming refers to the victims of crime being accused of being at fault for what someone did to them, often but not always because they are part of a societal outgroup.

> The whole sales pitch of GPT-4 is a do-what-I-mean interface, and it clearly did not do what they meant, or what any reasonable hacker would expect them to mean.

It's clearly marked as experimental, people are repeatedly told not to rely on it (ie when they login each time, and often by the ai itself), and there have now been a year+ of very public examples of various AIs getting things hilariously wrong.

From the login warning:

> While we have safeguards in place, the system may occasionally generate incorrect or misleading information and produce offensive or biased content. It is not intended to give advice.

Any "reasonable hacker" would extrapolate from that, that giving it access to an API could lead it to do unexpected things.

Any "reasonable hacker" would contain it within a sandboxed project.

Etc etc.

It's no more victim blaming than if someone ignored every safety regulation and got themselves hurt or killed. You are being told these things are beta, can produce incorrect results and then in every Convo OP is feeding an option for 'Do things with GitHub' context to the model.

Of course it's going to bias to using GitHub especially after he already used one plugin

Reply on news.ycombinator.com