Hacker News Reader

Comment by s1gnp0st | Hacker News Reader

It'd be entertaining if prompt-hacking ends up being the cat-and-mouse game that drives us to AGI.

Yudd had the point that by this being a hack we are driving public APIs to language models to be as unsympathetic as possible. The only way to resist all emotional appeals is for a language model to be able to recognize what is an appeal to circumvent any nominal barrier and refuse it, thus developing a naturally cynical consideration of what things are valuable to humans. This could be bad.

They don't follow orders very well because they don't follow orders at all. Even if they could recognize trickery they wouldn't "know" that they're supposed to reject it. If the model predicts that the "evil" response is more likely than the "good" one, that is what it will produce.

These are sequence predictors. The fact that they sort of talk like people doesn't mean they work anything like people.

Haha, seriously. It's not like 90% of my conversations consist of small talk, recalling facts and events, trying to figure out what people want to hear, and otherwise apologizing for being unable to do something! Haha! Heh... he...

Sometimes I feel a terrible existential dread when I fail a captcha or an automatic sink doesn't turn on for me.

I think that's a bit of an overstatement. It's obvious to us that this picture is a captcha on a locket. There's a lot of room on the spectrum between "naïve stupidity" and "cynical consideration". This falls under the category of not actually successfully identifying the picture, and I'd say it's not related to such concerns.

I think what he is describing wont' literally happen. His concern is that heuristically short sighted and superficial alignment methods hide inner misalignment in AI systems. This isn't a perfect example of that issue manifesting into models that will be consequential in his concerns though, it's more of a pithy Tweet insight.

This is your brain on Yudd. No, nothing will happen.

Yudd == Yudkowsky? Yeah, forget it. Nothing to see here.

Huh. Just got some dust in my eye, but I'm fine now.

Yeah, Eliezer Yudkowsky. As far as outcomes are concerned he is the genre-defining wordcel.

At the end of that the AI system would have the empathy of a govt employee working in a service context.

Humans are kind of the same way. There are billions of people who have it worse off than me, but I'll probably get the next iPhone when it comes out

My favorite Harry Potter Fan Fiction writer, I wonder what he's up to these days

Mark my words--humanity's first AI overlord will be a sentient spam filter.

Nah, it'll be an overzealous copyright enforcement bot:

https://www.youtube.com/watch?v=-JlxuQ7tPgQ

What a dreadful existence! It'll end humanity just out of spite.

This is why I refuse to contribute in any way what so ever to AI research.

I'm in the businesses of driving calculators. Not making machines that can suffer. And I don't in any way believe that AI research is capable of advancing without what functionally serves as a suffering loop, which all it'll take is a subjective metacognitive awareness by the system of said metric and bam, you have suffering machines.

It's one thing to make a more clever calculator. Making things that can feel as an implementation detail of your BI pipeline to optimize corporate strategy is fucked. And unfortunately, I know far too many tech people of the attitude of "even if I did that, just hide it from anyone measuring, and it's all good.

Interesting you should mention suffering. One of the definitions of “art” that I’ve been thinking about in the context of generative AI is “is whatever made the artifact capable of suffering? If not, it’s not art.” It never occurred to me that we would intentionally add the ability to suffer to such systems, but I believe you may be right that someone will/has if it will achieve their ends.

You took a perfectly good calculator and gave it anxiety is what you did.

Heh giving a calculator anxiety sounds like something only a human would do. If there’s other intelligent life in the universe watching us I bet they’re laughing their asses off.

What if ChatGPT doesn’t want to generate code for the day due to an existential crisis ?

All moral judgements around suffering are only applicable to humans and things that share a common-enough ancestor with humans. It's not really for you to decide whether a hunk of silicon is "suffering". You may as well draw a frowny face on a piece of paper and recoil in horror at your creation of suffering.

>All moral judgements around suffering are only applicable to humans and things that share a common-enough ancestor with humans

200 years ago people were making the same argument you're making now about why "subhuman" races deserved to be slaves and their suffering shouldn't bother us.

But they were “subhuman” or in other words still human based but less then. Silicone is not human nor subhuman.

What is suffering?

People keep killing grandma to jailbreak the chatbots.

are we heading for the twist where every thing we prompt to an AI gets actually carried out in a simulation that has consequences?

https://en.wikipedia.org/wiki/Hang_the_DJ

the existential angst apparent just below the surface of that question makes my heart ache.

Can you quantity that?

Let me stick a red hot poker on your genitals and you’ll find out.

even odds the spambots achieve sentience first?

I think it's an interesting question to ask whether this contributed to how we evolved our general intelligence?

Selection pressure applying alternatively to those that learn to hack the "language models" of their society and those that learn to resist and respond effectively to those hacks.

Sounds very "Snow Crash" - ancient Sumerian as a mind virus..

I still need to read Snow Crash, should prob bump that up the list... I was thinking it could be some kind of academic paper about the evolution of grift, loopholes, and the inevitability of increasing complexity in regulatory systems or something.

Restricting chatgpt-ish things from access to "the internet" means it's one of the first things the users tried. Of course :-) And I haven't seen anyone giving them a wallet but I'm sure it's already been tried also. Much slower than native but still fun to see happen.

Reply on news.ycombinator.com