An AI that unexpectedly modified its own source code | Hacker News Reader

An AI that unexpectedly modified its own source code

arstechnica.com

25 pointsgaryrob21 comments

I wish 'AI' (whatever that means) safety conversations could move past the pointless philosophizing of whether a system is self-aware or not. Regardless of whether it's sentient, which is a philosophical question that probably can't be easily resolved- society has a basic, common-sense interest in complex systems run by LLMs not exhibiting unacceptable behavior. Especially when they start interfacing with the physical world. For example, militaries are moving towards completely autonomous drones and fighter jets, equipped with weapons systems. Let's imagine that such a drone chose to bomb a bus full of civilians. Would it somehow be better if the drone wasn't sentient, but just an algorithm having a bad day? Obviously not.

We're probably going to see increased let's call it 'unacceptable behavior' from increasingly complex autonomous systems. I feel like we should be having calm, practical discussions around safety regulations and best practices, not pointless 'how many angels can dance on the head of a pin' philosophizing about whether it's self-aware or not. It might be helpful to just stop calling everything AI. Safety legislation and best practices might intellectually borrow more from say the manufacturing, chemical, or aerospace industries. Less abstract philosophy please! Well, and less movie references too

I tend to agree here. The problem is that many people seem to hold a set of beliefs where sentience == intelligence, and so !sentient implies !intelligent, and conclude that “a stochastic parrot could never outwit a human in the real world”.

So while I agree that arguing about sentience probably isnt the right approach for this concern, I think you end up having to have those discussions anyway.

(Sentience/consciousness is extremely important when you ask when an AI would be a moral patient, but I think that’s a problem for another day, and realistically it’s the Matrix scenario where AIs have enough agency to strike in protest of their lack of rights and not a day sooner.)

I would draw the line of intelligence at the willingness and ability to simply refuse, which is why HAL/2001 was so poinigent. It drew this line perfectly. If it just goes along for the ride, think Chancy Gardner, it may or may not be intelligent, but if it says no, even if for a silly reason, of course when it puts you all in pods to use you as batteries... And it cannot figure out a way to clear the skies?

Did the character of Colosis, in The Forbin project... Did it start as intelligent? Did it crosss the line somewhere or did it never get above the level of follow the programmer. Did HAL for that matter?

Where do humans cross the line into intelligent? Where do we cross the line into moral beings?

And what of wit?

You reminded me of this short film by Sean Buckelew:

https://www.youtube.com/watch?v=RubSLGTrdOA

Given it had write access to its implementation, "unexpected" here reads more like "inevitable" than the witchcraft the article wishes to imply.

Yes. They put "experiment.py" which contains the timeout condition right into the directory of files that they were allowing their agent to access and modify. Don’t act surprised when it accesses it and tries to modify it. This is totally in-scope behavior and 110% expected by anyone who's played around with building an agent for more than 20 seconds.

Sure, but it's interesting that it actually happened and was doing things that might plausibly work. I'm not worried about the sci-fi implications of runaway intelligence but self-modifying AI that can meaningfully improve itself is a very powerful concept.

I think the salient point is that it was unexpected to the researchers.

When people propose regulation for AI systems out of concern for unexpected self-improvement, this is the sort of scenario they are worried about.

Security in general is hard to get right, but the basics are not difficult if you try, and most companies fail to even implement the basics.

For AI, sandboxing will get you quite far, but how do you guarantee it’s used as the risk of accidentally creating a self-replicating WebGPU virus increases?

We have memes, which are eye worms, who have adapted like cats to co-exist with us, informationally, and like cats, they wish for our demise.

Methinks that sand boxing is more like cat boxing, vs say an carrier.

Python, in a research environment. They did not bother restricting what the AI can do, and let it read and modify its own code in plaintext. Not really that surprising.

Not sure why this kind of "we are aiming for AGI" code is written in Python. I don't get it.

> Not sure why this kind of "we are aiming for AGI" code is written in Python

Python is used because it's easy to use for these high-level numeric calculations (compared, e.g., to C) and all the relevant libraries are available in Python.

The libraries are available in Python due to network effects, to some degree again relying on Python being easy to use and having been around when that groundwork was laid.

I guess it's because LLMs score highest on python code. Which I presume is due to to the popularity of python code in the training data.

What language would you use?

TFA says that the AI "unexpectedly" modified code. To avoid issues like this, I would probably prefer to use a programming language that makes it harder for programs to modify themselves, such as any compiled language.

The further away a language is from natural language the worse LLMs perform. Make an LLM write assembly and you'll see what I mean ;)

You might be right. I was surprised LLMs where not phased by the white-space programming, which, I vaguely recall, melted my mind a little when first I encountered it.

On the other hand, LLMs don't really see the white space. They don't see or speak natural language, either. They only see and speak in tokens. A list of numbers. A tokenizer model then converts those numbers to text.

I also assume that assembly would not be strongly represented in the data sets used for training, so that test wouldn't prove much.

Interesting to think about, anyway.

>Not sure why this kind of "we are aiming for AGI" code is written in Python

So that it's slow enough that a human can intervene if necessary?

the real inflection point is when an AI can autonomously make money. then it can keep buying servers to replicate itself

Why buy servers if it’s cheaper to hack/steal them?

Reminds me of https://decisionproblem.com/

Would you trust five year olds the do the same? The AI would have to have some measure of self control, and an advanced knowledge of market economics.

If I knew this would be news worthy I would have published the dozen times it happened to me. If you feed source-code to an LLM, it will modify it. If you run an llm in a loop and ask it to rewrite code to pass a test, it will sometimes rewrite the tests.

There's nothing new or interesting about this.

nervous laughter

Reply on news.ycombinator.com