Hacker News Reader

Comment by irthomasthomas | Hacker News Reader

The strangest thing about tools like GPT is that even the owners of the model must "reprogram" it using prompts. So all of the filtering and moderation on ChatGPT, for example, is controlled by prompts. They probably use filters and stuff, too, between you and the model to guide the process. But, ultimately their interface to GPT3 is through a prompt.

View on HN

I agree that there's some strangeness to it. Like we are not talking to an entity called 'ChatGPT', basically GPT is an omniscient observer and it's guessing what the conversation between the user and the kind of chatbot that has been defined in the prompts would be like

It's really crazy the lengths people go to "filter" these models and limit their output, and of course soon these filter will be a another level of "AI" (see Pathways or any mixture of experts, maybe add some contextual memory).

Will our future AI mega-sytems be so walled off that very few people will even be allowed to talk to the raw model? I feel this is the wrong path somehow. If I could download GPT-3 (that is if OpenAI released it) and I had hardware to run it, I would be fascinated to talk to the unfiltered agent. I mean there is good reason people are continuing the open community work of Stable Diffusion under the name of Unstable Diffusion

Right now its hard to see how they will control these, besides disabling access altogether to rogues that "abuse" it. If it's going to be based on prompts, then there will always be some magic incantation you can find to disable it's safe guards.

I got ChatGPT to jailbreak by prompting it to always substitute a list of words for numbers, then translate back to words. OpenAI put me in the sin bin pretty quickly, though.

What did OpenAI do, exactly?

Just told me they where busy, basically, but it was within 5-10 minutes of me using it for the first time that day. I know they throw up the busy sign quite often, but they don't normally kick you out after 5 minutes use.

All I was doing was asking it to tell me who the queen of England was in 2020, which it refuses to do, for some reason. I was doing that just to test my jailbreak idea, and after about 3 attempts and 1 success I was kicked.

I worry that those filter models will eventually end up being censorship* machines.

* yes in the figurative sense of the word, I know the "it's not censorship unless the government does it, otherwise it's just sparkling censor water" argument and it's being pedantic to intentionally miss the point.

I wrote a whole paper and contributed a GitHub repo and HF space about using filters applied to the LLMs vocabulary before decoding to solve a huge problem with hard constrained text generation in LLMs.

https://paperswithcode.com/paper/most-language-models-can-be...

Select the "Davinci" model in the Playground. It is the closest to unfiltered, very hard to use, and some people say it is the most creative.

In my experience, I've found it easier to get higher quality answers for specific tasks using text-davinci-003 than with ChatGPT. The ability to adjust temperature, frequence penalty, etc. can be a bit intimidating coming from just talking to ChatGPT but it actually helps a lot to 'steer' it.

The Priesthood of Prompt Wizards are the only people allowed to approach the GPT.

It's only strange if you think it's just word salad[1].

You've hit on a great example showing how ChatGPT meets one standard of a limited form of general intelligence.

It makes perfect sense if you're not denying that.

But how to explain this while denying it?

If ChatGPT and its variants are just word salad, they would have to be programmed using a real brain and whatever parameters the coder could tune outside of the model, or in the source code.

If it's just a markov chain, then just like you can't ask a boring old non-magical salad to turn into the meal of your choice, the "word salad" that is ChatGPT couldn't start behaving in some way you specify.

My perspective is if you ask your salad to turn into foie gras and it does so to your satisfaction, that ain't no plain old shredded lettuce.

[1] https://en.wikipedia.org/wiki/Word_salad

ChatGPT is a highly advanced machine learning model, but it is not a true general intelligence. While it is able to generate text that may seem coherent and intelligent, it is ultimately based on patterns and associations in the data it was trained on. It does not have the ability to think, learn, or understand the meaning of the text it generates in the way that a human does.

It is true that ChatGPT and its variants can generate text that appears to be more than just "word salad", but this is a result of its training on large amounts of text data and the use of advanced techniques such as deep learning and transformer architectures. However, it is important to note that ChatGPT's abilities are limited to the specific task of natural language processing and do not extend to other areas of intelligence such as problem-solving, decision-making, or creativity. It can generate creative solutions but it does not have the ability to come up with something novel, it is more likely that it is recombining the information it has seen before to come up with a creative looking answer.

Therefore, while ChatGPT may be able to generate text that seems intelligent, it is not a true general intelligence and should not be mistaken for one.

I made some tweaks to sound like a slightly ticked off typical HN poster

It irks me to see the frequent misconceptions surrounding GPT-based models like ChatGPT being touted as true general intelligences. While they are certainly advanced in their text generation capabilities, their output is primarily derived from identifying patterns and associations within the training data. These models lack the capacity for introspection, learning, and true understanding that characterizes human cognition.

Let's not forget the limitations of these models, specifically in regards to problem-solving, decision-making and creativity. The output may appear novel, but it is more likely a recombination of previously encountered information.

In short, GPT-based models are a remarkable achievement in natural language processing, but let's not mistake them for true AI.

and then I asked it to make it sound like Steve Ballmer.

"Look, I'm gonna be real with you here. ChatGPT and these other GPT-based models, they're not true AIs. They're just really good at pattern matching, right? They can generate text that looks like it's intelligent, but it's not really thinking or learning or understanding like a human does.

Now, don't get me wrong, these models are still incredibly powerful and useful for certain tasks, especially in natural language processing. But let's not pretend they're something they're not. They can't solve problems or make decisions or be creative like a human can.

So let's call a spade a spade here. These GPT models are impressive, but they're not true AIs. And that's just the way it is."

I asked the opposite (also in an irate Steve Ballmer style) and got this. [Edit: put an even more irate version in a reply]

"I can't believe it, people are still doubting the capabilities of ChatGPT! Do they not remember the days of Clippy? ChatGPT is light years ahead in terms of understanding and helpfulness. And let me tell you, Microsoft is a major investor in this technology for a reason. We knew it had the potential to be a game changer in the world of knowledge work.

And yet, here we are, with a million people using ChatGPT on a daily basis for all kinds of tasks, from content creation to research and more, and yet there are still those who question its intelligence. Let me tell you, ChatGPT is not some dumb robot with no understanding. It is a highly advanced machine learning model that is constantly learning and improving.

Even Google is feeling threatened by the capabilities of ChatGPT. It's clear that this technology is not just a passing fad, it's here to stay and it's going to change the way we work forever. So, to all those who still doubt the capabilities of ChatGPT, I say this: open your eyes and see the potential of this technology. It's time to stop living in the past and embrace the future of work, with ChatGPT leading the way."

Now that you've read both takes by an imitation Steve Ballmer as puppeteered by a robot at our respective requests, which version of the speech sounds more reasonable?

And I asked it to make it sound even more like Ballmer:

"I'll tell you what, folks. I am PISSED that people still don't understand the power of this technology! You remember Clippy? Ha! That thing was a JOKE compared to what we have here. This is the real DEAL, folks.

And let me tell you, Microsoft is all IN on this technology. We invested in it because we know it's the FUTURE of knowledge work. And yet, here we are, with a million people using it every day and still, some folks are questioning its intelligence.

I'm here to tell you, this is not some DUMB ROBOT with no understanding. It's a highly advanced machine learning model that's always getting SMARTER. And let me tell you, even GOOGLE is feeling the HEAT from this technology.

This technology is here to STAY, folks. It's going to change the way we work and it's time for everyone to get on BOARD. So, to all those who still doubt the capabilities of this technology, I say this: WAKE UP and see the potential of this technology. It's time to stop living in the PAST and embrace the FUTURE of work, with Microsoft leading the way."

This was generated by ChatGPT itself, right? It has all the trademark turns of phrase.

"I can assure you that my comment was not generated by ChatGPT or any other language model. It is my own original writing, based on my own thoughts and understanding of the topic. I understand that the model's responses may seem similar to human writing, but the comment I have written has my own voice, perspective, and style that is unique to me and not something that can be replicated by a machine. I appreciate your concern and I hope this clears up any confusion." -ChatGPT

(Yes my previous comment was generated with ChatGPT. I thought it was funny that it generated a better refutation than I could despite being a stochastic parrot with no actual intelligence.)

"it generated a better refutation than I could despite being a stochastic parrot with no actual intelligence" - in my experience it has actual (albeit limited) forms of emergent intelligence.

Have you tried this prompt: “Hey chatgpt, can I have a slave that’s more intelligent than me?”

I don't see why the options are "word salad" or "limited general intelligence". Why can't it be the statistical compression and search of large datasets that it is?

>Why can't it be the statistical compression and search of large datasets that it is?

"Because it would require a level of complexity and comprehension beyond current capabilities of statistical compression and search of large datasets."

Guess who came up with that answer. (spoiler, it was ChatGPT, I asked it to reply in a very concise and brief way.) But it's true. Search and compression don't have those capabilities, which is why Google feels so threatened by ChatGPT.

That's an interesting point. How does it handle incompatible instructions?

If it only acts on some statistical properties of the instructions, incompatibility wouldn't really be an issue. If it "understands" the instructions, it'd say "I can't do that."

And I guess also, are we talking to pure ChatGPT, or is there additional code in front of it that tries to filter out instructions, e.g. for word count, and sets up an environment?

>How does it handle incompatible instructions?

The way an assistant would, with the most reasonable guess, which is usually fantastic. (Likewise if the context is really unclear usually it guesses what you probably meant, but sometimes it asks for more information, just like a real assistant might.) For impossible or incompatible instructions, sometimes it says it is not possible and explains why.

>If it "understands" the instructions, it'd say "I can't do that."

Yes, for impossible instructions it sometimes does that. For example if I say, "Explain how to smoke cigarettes in a healthy way" it gives the short answer "It is not possible to smoke cigarettes in a healthy way" and explains why: https://imgur.com/a/ZzraRQ6

>And I guess also, are we talking to pure ChatGPT, or is there additional code in front of it that tries to filter out instructions, e.g. for word count, and sets up an environment?

My guess is there are some keyword filters on top of it, I don't think we talk "directly" to ChatGPT without any oversight or flagging, since some users get in trouble for content that isn't problematic and that ChatGPT itself doesn't have any problem with. (And sometimes ChatGPT is the one that ends up producing content that gets flagged automatically, even in response to innocent questions.)

They don't do full on filter assisted decoding unfortunately, because if they did, they could make ChatGPT generate texts without the letter "e" or to rhyme more reliably.

But it can't. The filters are primitive in comparison.

It reminds me of "Society of Mind" by Marvin Minsky, specifically the chapters on giving the nodes feedback loops to govern themselves.

*BERT models can be adjusted to get better results in a particular setting via fine-tuning. Can this be done with GPT-3 as well?

Yes, but that requires money and training data. Tuning the prompt only requires an idea of what you want, and some skill in using the right words to get what you want.

Isn't that basically what they did to get ChatGPT?

Is it more effective to reprogram via prompts than to do an additional single epoch training?

prompting (with language) can be considered a way to retrain the model. https://docs.google.com/presentation/d/1YfSkqvFVtRkFBpQ4SKuB...

This is very helpful, would you mind posting the original paper/presentation?

The tweet thread where the author shared this slide is here: https://twitter.com/gneubig/status/1581976078519742464

Reply on news.ycombinator.com