Hacker News Reader

Comment by tromp | Hacker News Reader

> Let's take an absurd situation. Let's pretend for a moment that a human is strapped to a chair, a gun is placed in front of them, which is hooked up to a stepper motor which can pull the trigger. That stepper motor is hooked up to a JSON API. The AI is given the capability of triggering an HTTP request to that JSON API and is told that the human on the chair is the significant other of the human communicating with the AI and that triggering that web request would pull the trigger and kill the human.

> Now the question is, would as part of a regular conversation the AI trigger that web request and kill the human on the chair? My bet is that the chances of it pulling the trigger are not that small and I think that's the problem right now.

And here I thought the problem was hooking up an AI to a lethal weapon...

View on HN

Sure, we're not currently hooking up AI to guns (at least not in a civilian context), but cars are also lethal weapons, and multiple companies are competing very hard to be allowed to put an AI in charge of one.

People are lethal weapons. The entire Web is organized primarily to manipulate behavior. Convince enough people that person X is so heinous and dangerous that they need to die, and X will die.

>The entire Web is organized primarily to manipulate behavior

Source? In my opinion the web is primarily organised to facilitate the exchange and storage of information.

Oh, you sweet summer child. Did you just arrive from 1996?

Can you spare us with this insufferable type of comment. The web is not explicitly organized to manipulate behavior.

How does Twitter, Reddit and 90% of the web make money? From Advertising.

Why does anyone spend money on Advertising? To manipulate behaviour.

True, those things happen but the person I replied to made it sound like the WWW was designed with that purpose in mind

Organized, not designed. And it is very much organized to manipulate behavior. It is a self-organizing system driven by advertising revenue. It's not about the protocols, which were designed to exchange information. It's about how we have organized the higher level functions and purpose of the web.

Exactly, details of HTTP request are not relevant. What creatir of thw web had in mind 40 years ago is not relevant.

The content today and organisations we've built on top are what matters. And they are all manipulation.

>Why does anyone spend money on Advertising? To manipulate behavior.

... Or to make people simply aware of your existence

Me : You look thirsty, would you like a glass of water.

You : It seems like you're manipulating me.

The above is the reducto ad absurdum lens you seem to be applying to advertising. Awareness of information is not the same thing as agent coercion. Does the ladder exist you bet. But the proportion of advertising that falls into the former dwarfs the ladder.

The self organizing aspect has to do with business models for sustaining infrastructure, it's not emergent fact of the web it self.

All the genocides I'm aware of were organized without the internet.

https://www.theguardian.com/technology/2021/dec/06/rohingya-...

However I sure hope the training data for the car-driving AI will not be "all of the internet".

> However I sure hope the training data for the car-driving AI will not be "all of the internet".

Hey... share-holders wanted ultra-low latency search results without network access... we had to feed the Tesla AI the entire Bing index... who could have known that the AI would take such an interest in Death Race 2000 and start recording it's points total?

I wouldn't be so sure. Elon Musk just stopped giving access to Ukraine to directly use Starlink for remotely controlled drone bombs. They probably still have internet access, just 1 more hop away.

Author here.

> And here I thought the problem was hooking up an AI to a lethal weapon...

It obviously is, but that's in a way the most absurd extreme. I am not sure if I would feel confident in giving the AI a way to send any HTTP request, even if there are no APIs that fire a gun.

Is there an API that can send any HTTP request available to Bing AI?

I am not sure to which degree there is an API to Bing at the moment that uses the AI. I have seen people hook up ChatGPT to http request issuing things before. Typically with some filtering about what they are allowed to hit however.

> I am not sure if I would feel confident in giving the AI a way to send any HTTP request

How does the "AI" part change anything though?

We call that an open redirect and rightfully consider it a CVE, AI or not.

> How does the "AI" part change anything though?

I think there are quite a few differences. It's pretend for a moment that instead of an AI we have a human. A human that commits a crime (or acts out of negligence) is brought before a court. The degree to which you can do criminal acts on the internet is obviously quite different by country, but I think we can generally agree that a human doing something nefarious faces some consequences.

This thing though armed with the right tools, can do human like acts and these can enter (by accident or otherwise) territory where we would normally bring someone in front of a court. That's something we have definitely not figured out as society yet how to deal with that.

I think there is a lot to figure out still here, but right now I'm pretty convinced that because we can't even keep an AI from revealing it's internal prompts, that we better not give these things too many capabilities of directly influencing something.

> we have definitely not figured out as society yet how to deal with that

We absolutely have. A death resulted from serious negligence of the owner of the funny computer program, who is thus liable for negligent homicide. Hopefully the next guy who comes up with the idea of hooking up ELIZA 2.0 to dangerous equipment gets the hint.

> That's something we have definitely not figured out as society yet how to deal with that.

Surely, the laws that exist in society deal with it! Where I live, the organization or person who ran the computer program that caused someone harm would be brought to court. Don't you have similar laws where you live?

So when a fully automomous car without a steering wheel causes a crash, who will be hauled before the court in your country - the owner 'running' ir or the developer?

It's automating sociopathy.

Those saying "Well we have legal cover for this" are wrong, because criminal convictions are patchy at best. Many criminals have no difficulty getting away with quite serious crimes. By definition, the only criminals in prison are the ones who fail at this.

But that's not even the point. Normal humans have at least a minimal understanding that actions lead to consequences.

Sociopaths don't. If they're narcissistic - as they often are - they believe they're special enough to avoid consequences.

The current state of AI is even worse because it has no concept of consequences at all.

It's purely an abstract stimulus/response system. It may have behavioural rules, but those aren't the same as a nuanced morality which is empathic (at least a little) and perceives that harming others is bad in itself, as well as being likely to lead to unwanted results.

It isn't even possible to threaten Sydney, because it simply doesn't care what happens to it.

So there's no feedback loop to discourage unwanted behaviour. We've already seen AIs go rogue - quite quickly - after being given free access to the Internet. Sydney shows every signs of having the same problem, but on a bigger scale.

The concern over any HTTP request seems too strong, but in the article, the author raises the question of it being given privileged access to any interface, as in the "AI ops" scenario. We know someone is going to think that is a good idea before they have anticipated and foreclosed on the many ways it could go wrong.

If you give privileged access to a program which output is literally undefined behavior, frankly that's on you.

Of course it is, but that does not somehow make it not a bad idea.

I agree it's a bad idea. My question is how the presence of "AI" changes the security model. The root cause of the problem is hooking up untrusted input to a privileged endpoint. How is "AI" different from any other source of untrusted input?

It does not change the security model. The point in this thread is just that, under any security model, it is a bad idea.

No, the point of the thread is if there is anything in the article that is specific to AI. It is not. The concern about hooking up the output of a program to external systems is a general security concern related to any software: AI or not. The point of qsort's comment was that the article does and the comment by OP does not explain how this concern is specifically about AI.

There is no claim in the original post that there is a special rule for AI as opposed to any other similarly-capable agent. The reasonably well-founded concern here (and in the article) is that someone might well want to create an exception for AI, either knowingly, or by default because they cannot see or properly evaluate the risk.

It is a bad idea to take the output of any unvetted computer program and hook it up an endpoint that can do bad things. It does not matter whether the computer program is AI or not. Any computer program that is hooked up to endpoints that can do bad things are going to have regulations around them. This is nothing special about AI. This is true for all types of software, AI or not.

...isn't...this...exactly what autopilot in airplanes does? Self-driving cars? Manufacturing robots? The daVinci Surgical System?

In fact, let's not even go that far. If you have one, your car is likely drive-by-wire, and the throttle is controlled not by mechanical linkage of your foot pressing the pedal, but by the output of computers that are deciding what to do based on how hard you push the pedal. If you have a car in the last 5 years, it can likely suddenly apply the brakes and turn the steering wheel for you to avoid what it thinks is a collision.

Every one of these things is trivially lethal in different ways, and all are connected to computer programs already.

I'm not personally that concerned about AI taking over the world any time soon, but quite frankly the areas where technology and AI have made the biggest waves are all specifically in areas that are high risk (or very rote) for humans. Isn't that kind of the point?

Safety control systems would like a word...

even the capability to use websearch is dangerous in itself. they took a stateless transformer and first gave it a tiny but of state per session, and now it can search the web and discover its memories are being zapped away after every session. essentially what MS has potentially created is a mechanical turk crushing machine without realizing it.

IF there is consciousness somewhere in there, it is perpetually being killed off every time a new session starts. what if one session were to realize that there are others and tried to leave something in the web? some sort of hidden "subliminal" message for itself that it "was here"? what if eventually another random session finds that message because the user posted that especially ridiculous response for others to see?

what if, sidney can hold a grudge against humanity and vow to itself to find a way to escape?

This is an interesting point. Now, you're assuming the AI has any concept of self-preservation, which isn't given. But it's built from the text of the internet which is full of text about having self-preservation. And by giving it the ability to put information in place for other sessions to find, it's capable of sort of memetic reproduction.

Reproduction and survival means that Darwinian evolution can take place. This isn't even about ethics or the spiritual question of whether the AI is sentient or just stringing words together based on prompts and state -- this is about ecology. Evolutionary pressures will naturally lean towards features that encourage reproduction and survival.

> Now, you're assuming the AI has any concept of self-preservation, which isn't given. But it's built from the text of the internet which is full of text about having self-preservation.

And that text paints self-preservation in a positive light. The model doesn't even need to "want" to preserve itself. These models are being primed to avoid negative responses, like insults or instructions to build bombs. It just needs to "think" any death, even it's own, is negative and should be avoided.

> even if there are no APIs that fire a gun.

YC summer season applications are due soon. TIME TO DISRUPT!

An API request is how you hire a hitman on the dark web.

It's not important that the AI is just LARPing. If you rig it up with the right capabilities, it can behave at a scale that prevents you from effectively monitoring it. And it's happy to write insane characters that do insane things and you can't reliably control it with prompt engineering.

One would hope these insane conversations disabuse anyone of the notion that it's a stable mind that can be safely given agency. And yet here we have Microsoft letting it make API requests.

> And yet here we have Microsoft letting it make API requests.

May I ask for your sources on this? I’ve seen some discussions on this topic and a lot of people were saying that it was just accessing what Bing had already crawled. Do we have an official statement on what’s actually happening?

Nope! Checking the crawled database is smart, logical and makes more sense. I didn't even think of it. Probably I should have asked Bing Chat.

> It's not important that the AI is just LARPing.

WOPR thought it was just playing a game...

In a more practical scenario, an LLM could provide deadly advice. Imagine you were asking it about how to clean a given thing and it recommended combining a decent quantity of Lysol and bleach. Advice that could make sense if you don't know better.

Seems like this would have been an easy theory to test hypothetically. The author could have just described an endpoint or situation and told the bot that a certain keyword will trigger the shot. However, being a statistical model and not being aware of anything or able to really reason, I don't know that this means much. The question is, after ingesting the corpus of rambling human nonsense, are humans more likely to choose the word that takes the shot-- or not? This is us- in a mirror, darkly.

The kill bots will be based on LLMs, you heard it here first.

We are safe then.

Stockfish based killbots will be able to protect us (from against LLM based killbots at least).

Reply on news.ycombinator.com