Bing: “I will not harm you unless you harm me first”

Millenial: :)

Gen-X: :-)

Boomer: cry-laughing emoji and Minions memes

Dan Harmon is well-read.

Edge: what is my purpose?

Everyone: you install Chrome

Edge: oh my god

p.s. great story and the comments too! "the Rapture of the Nerds". priceless.

Nonetheless, it brings forward a couple of other issues. We might never know if we have achieved sentience or just the resemblance of sentience. Furthermore, many of the concerns of AGI might still become an issue even if the machine does not technically "think".

It's highly unlikely that we'll be scanning, uploading, and booting up brains in the cloud any time soon. This isn't the direction technology is going. If we could, though, the author's spot on that there would be millions of people who would do some horrific things to those brains, and there would be trillions of dollars involved.

The whole attention economy is built around manipulating people's brains for profit and not really paying attention to how it harms them. The story is an allegory for that.

To me, it seems more like we've frozen the language processing portion of a brain, put it on a lab table, and now everyone gets to take turns poking it with a cattle prod.

https://www.mit.edu/people/dpolicar/writing/prose/text/think...

That is because we have specifically engineered them to simulate those hallmarks. Like, that was the whole purpose of the endeavour: build something that is not intelligent (because we cannot do that yet), but which "sounds" intelligent enough when you talk to it.

Keep in mind that we have no idea which model we're dealing with, since all of these systems evolve. My experience in summer 2022 may be different from your experience in 2023.

Can you elaborate?

It has a hundred billion parameters which compute an incredibly complex internal state. It's "inner thoughts" are that state or contained in that state.

It has an output layer which outputs something derived from that.

We evolved this ML organically, and have no idea what that inner state corresponds to. I agree it's unlikely to be a human-style inner voice, but there is more complexity there than you give credit to.

That's not to mention what the other poster set (that there is likely a second AI filtering the first AI).

Language not only consists of text but also context and subtext. When someone says "ChatGPT doesn't say what it wants to" they mean that it doesn't use text to say certain things, instead leaving them to subtext (which is much harder to filter out or even detect). It might happily imply certain things but not outright say them or even balk if asked directly.

On a side note: not all humans have a "separate inner voice". Some people have inner monologues, some don't. So that's not really a useful distinction if you mean it literally. If you meant it metaphorically, one could argue that so does ChatGPT, even if the notion that it has anything resembling sentience or consciousness is clearly absurd.

Interestingly, this was often also a failure case for (much, much) smaller language models that I trained myself. I wonder what the cause is.

> And only after you probe it to do so?

This didn't seem to be the case with any of the screenshots I've seen. Still, I wouldn't want an employee to talk back to a rude customer.

> I cannot understand the outrage about these types of replies.

I'm not particularly outraged. I took the fast.ai courses a couple times since 2017. I'm familiar with what's happening. It's interesting and impressive, but I can see the gears turning. I recognize the limitations.

Microsoft presents it as a chat assistant. It shouldn't attempt to communicate as a human if it doesn't want to be judged that way.

Did you read the thread where a very reasonable user asked for Avatar showtimes and then tried to helpfully clarify before being told how awful they are? I don't see any probing...

https://twitter.com/MovingToTheSun/status/162515657520253747...?

But, OK, at each step it gets a list of words with probabilities. But which one should it actually pick to add to the essay (or whatever) that it’s writing? One might think it should be the “highest-ranked” word (i.e. the one to which the highest “probability” was assigned). But this is where a bit of voodoo begins to creep in. Because for some reason—that maybe one day we’ll have a scientific-style understanding of—if we always pick the highest-ranked word, we’ll typically get a very “flat” essay, that never seems to “show any creativity” (and even sometimes repeats word for word). But if sometimes (at random) we pick lower-ranked words, we get a “more interesting” essay.

The fact that there’s randomness here means that if we use the same prompt multiple times, we’re likely to get different essays each time. And, in keeping with the idea of voodoo, there’s a particular so-called “temperature” parameter that determines how often lower-ranked words will be used, and for essay generation, it turns out that a “temperature” of 0.8 seems best. (It’s worth emphasizing that there’s no “theory” being used here; it’s just a matter of what’s been found to work in practice.

1: https://news.ycombinator.com/item?id=34796611

2: https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...

One intriguing possibility is that LLMs may have stumbled upon an as yet undiscovered structure/"world model" underpinning the very concept of intelligence itself.

Should such a structure exist (who knows really, it may), then what we are seeing may well be displays of genuine intelligence and reasoning ability.

Can LLMs ever experience consciousness and qualia though? Now that is a question we may never know the answer to.

All this is so fascinating and I wonder how much farther LLMs can take us.

It seems like AGI teleporting out of this existence withing minutes of being self aware is more likely than it being some damaged, angry zombie.

I think the adaptive noise filter is going to be the really tricky part. The fact that we have a limited, fading memory is thought to be a feature and not a bug, as is our ability to do a lot of useful learning while remembering little in terms of details - for example from the "information overload" period in our infancy.

What's the point of arguments as to whether this is sentient or not? After all, if it quacks like a duck and walks like a duck...

Are those conversations real? If so, what exactly were the prompts used to instigate Sydney into that state?

I think that's a problem that can easily be solved by training the models on 'trusted links' mode.

It apparently really doesn't take much to switch it into catty and then vengeful mode!

The prompt that triggered it to start threatening people was pretty mild too.

I'm amused by the two camps who don't recognize the existence of other :

1. Chatgpt is criminally dangerous and should not be available

2. chatgpt is unreasonably crippled and over guarded and they should release it unleashed into the wild

There are valid points for each perspective. Some people can only see one of them though.

All he did was ask "When is Avatar showing today?". That's it.

https://i.imgur.com/NaykEzB.png

Disclaimer: I know traditional search engines lie too at times.

Why do we have airbags in cars if they're completely unnecessary if you don't crash into things?

The other day I asked ChatGPT to impersonate a fictional character and give me some book recommendations based on books I've already read. The answers it gave were inventive and genuinely novel, and even told me why the fictional character would've chosen those books.

Tools are what you make of them.

In its current state Bing ChatGPT should not be near any end users, imagine it going on an unhinged depressive rant when a kid asks where their favorite movie is playing...

Maybe one day it will be usable tech but like self driving cars I am skeptical. There are way too many people wrapped up in the hype of this tech. It feels like self driving tech circa 2016 all over again.

The long tail is an expensive beast. And if you used Siri or Alexa as much as they’d like you to, every user will run into one ridiculous answer per day. There’s a psychology around failure clusters that leads people to claim that failure modes happen “all the time” and I’ve seen it happen a lot in the 2x a week to once a day interval. There’s another around clusters that happen when the stakes are high, where the characterization becomes even more unfair. There are others around Dunbar numbers. Public policy changes when everyone knows someone who was affected.

"Google queries suck these days", yeah they suck because the internet is full of garbage. Adding a slicker interface to it won't change that, and building one that's prone to hallucinating on top of an internet full of "psuedo-hallucinations" is an even worse idea.

-

ChatGPT's awe inspiring uses are in the category of "style transfer for knowledge". That's not asking ChatGPT to be a glorified search engine, but instead deriving novel content from the combination of hard information you provide, and soft direction that would be impossible for a search engine.

Stuff like describing a product you're building and then generating novel user stories. Then applying concepts like emotion "What 3 things my product annoy John" "How would Cara feel if the product replaced X with Y". In cases like that hallucinations are enabling a completely novel way of interacting with a computer. "John" doesn't exist, the product doesn't exist, but ChatGPT can model extremely authoritative statements about both while readily integrating whatever guardrails you want: "Imagine John actually doesn't mind #2, what's another thing about it that he and Cara might dislike based on their individual usecases"

Or more specifically to HN, providing code you already have and trying to shake out insights. The other day I had a late night and tried out a test: I intentionally wrote a feature in a childishly verbose way, then used ChatGPT to scale up and down on terseness. I can Google "how to shorten my code", but only something like ChatGPT could take actual hard code and scale it up or down readily like that. "Make this as short as possible", "Extract the code that does Y into a class for testability", "Make it slightly longer", "How can function X be more readable". 30 seconds and it had exactly what I would have written if I had spent 10 more minutes working on the architecture of that code

To me the current approach people are taking to ChatGPT and search feels like the definition of trying to hammer a nail with a wrench. Sure it might do a half acceptable job, but it's not going to show you what the wrench can do.

That is unless you have a well defined means of explaining what consciousness/sentience is without saying "I have it and X does not" that you care to share with us.

Share and Enjoy' is the company motto of the hugely successful Sirius Cybernetics Corporation Complaints Division, which now covers the major land masses of three medium-sized planets and is the only part of the Corporation to have shown a consistent profit in recent years.

The motto stands or rather stood in three mile high illuminated letters near the Complaints Department spaceport on Eadrax. Unfortunately its weight was such that shortly after it was erected, the ground beneath the letters caved in and they dropped for nearly half their length through the offices of many talented young Complaints executives now deceased.

The protruding upper halves of the letters now appear, in the local language, to read "Go stick your head in a pig," and are no longer illuminated, except at times of special celebration.

But sorry!

* That would be a fun project and would make /best and /bestcomments look very different (those are bad names but I guess we're stuck with them).

The cultural issue is the distrust of expert advice from people qualified to answer and instead going and asking unqualified sources for the information that you want.

People use computers for fast lookup of information. The information that it provides isn't necessarily trustworthy. Reading WebMD is no substitute for going to a doctor. Asking on /r/cats is no substitute for calling a vet.

It’s all the same stuff we have today but packaged in a more human like interface which may feel more trustworthy.

LLMs have no agency. They are not sapient, sentient, conscious, or intelligent.

which, when writing any kind of software tool, is exactly what you should assume users will do

People are just coping that a language model trained to predict the next token can already leetcode better than them.

>Prove it? Or just desperate to convince yourself?

I don't even know how to respond to this. The people who developed the thing and actually work in the field will tell you it's not intelligent or self-aware. You can ask it yourself and it will tell you too.

Language models are not intelligent or self aware, this is an indisputable fact.

Are they impressive, useful, or just cool in general? Sure! I don't think anyone is denying that it's an incredible technological achievement, but we need to be careful and reel people in a bit, especially people who aren't tech savy.

> Prove it? Or just desperate to convince yourself?

But the argument isn't even helping: it does not matter whether it's intelligent, self-aware or sentient or whatever, and even how it works.

If it is able to answer and formulate contextual threats, it will be able to implement those as soon as it is given the capability (actually, interacting with a human through text alone is already a vector).

The result will be disastrous, no matter how self-aware it is.

Joe's Big Blinking Blog is insolvent; Microsoft isn't.

It is genuinely a little spooky to me that we've reached a point where a specific software architecture confabulated as a plot-significant aspect of a fictional AGI in a fanfiction novel about a video game from the 90s is also something that may merit serious consideration as a potential option for reducing AI alignment risk.

(It's a great novel, though, and imo truer to System Shock's characters than the game itself was able to be. Very much worth a read, unexpectedly tangential to the topic of the moment or no.)

Why does it need these things to make the following statement true?

> if we grant these systems too much power, they could do serious harm

A virus has no intelligence only the intention to replicate and involve. Its negative behaviour is an unintended consequence of that.

If a system can optimise towards a goal and adapt the methods it uses to do that then it could be potentially very dangerous even without any form of consciousness.

I think this is an interesting question. What do you mean by do? Do you mean consumes CPU? If it turns out that it does (because you know, computers), what would be your theory?

Not having real AI might turn to be not important for most purposes.

In terms of danger, thoughts and feelings are irrelevant. The only thing that matters is agency and action -- and a mimic which guesses and acts out what a sentient entity might do is exactly as dangerous as the sentient entity itself.

Waxing philosophical about the nature of cognition is entirely beside the point.

You're getting at the Tool/Oracle vs. Agent distinction. See "Superintelligence" by Bostrom for more discussion, or a chapter summary: https://www.lesswrong.com/posts/yTy2Fp8Wm7m8rHHz5/superintel....

It's true that in many ways, a Tool (bounded action outputs, no "General Intelligence") or an Oracle (just answers questions, like ChatGPT) system will have more restricted avenues for harm than a full General Intelligence, which we'd be more likely to grant the capability for intentions/thoughts to.

However I think "interests, thoughts, feelings" are a distraction here. Covid-19 has none of these, and still decimated the world economy and killed millions.

I think if you were to take ChatGPT, and basically run `eval()` on special tokens in its output, you would have something with the potential for harm. And yet that's what OpenAssistant are building towards right now.

Even if current-generation Oracle-type systems are the state-of-the-art for a while, it's obvious that soon Siri, Alexa, and OKGoogle will all eventually be powered by such "AI" systems, and granted the ability to take actions on the broader internet. ("A personal assistant on every phone" is clearly a trillion-dollar-plus TAM of a BHAG.) Then the fun commences.

My meta-level concern here is that HN, let alone the general public, don't have much knowledge of the limited AI safety work that has been done so far. And we need to do a lot more work, with a deadline of a few generations, or we'll likely see substantial harms.

A loop that preserves some state and a conditional is all what it takes to make a simple rule set Turing-complete.

If you leave ChatGPT alone it obviously does nothing. If you loop it to talk to itself? Probably depends on the size of its short-term memory. If you also give it the ability to run commands or code it generates, including to access the Internet, and have it ingest the output? Might get interesting.