Hacker News Reader

Comment by tinco | Hacker News Reader

You had the right understanding in your first comment, but what was missing was the fine tuning. You are right that there aren't many documents on the web that are structured that way, so the raw model wouldn't be very effective on predicting the next token.

But since we know that it will complete a command when structured it cleverly, all we had to do to fine tune it is synthesize (generate) a bazillion examples of documents that actually have the exact structure of a system or an assistant being told to do something, and then doing it.

Because it's seen many documents like that (that don't exist on the internet, only on the drives of OpenAI engineers) it knows how to predict the next token.

It's just a trick though, on top of the most magic thing which is that somewhere in those 175 billion weights or whatever it has, there is a model of the world that's so good that it could be easily fine tuned to understand this new context that it is in.

View on HN

You’ve expressed this very well - Thank you.

I get that the fine tuning is done over documents which are generated to encourage the dialog format.

What I’m intrigued by is the way prompters choose to frame those documents. Because that is a choice. It’s a manufactured training set.

Using the ‘you are an ai chatbot’ style of prompting, in all the samples we generate and give to the model, text attributed to {:system} is a voice of god who tells {:assistant} who to be; {:assistant} acts in accordance with {:system}’s instructions, and {:user} is a wildcard whose behavior is unrestricted.

We’re training it by teaching it ‘there is a class of documents that transcribe the interactions between three entities, one of whom is obliged by its AI nature to follow the instructions of the system in order to serve the users’. I.e., sci-Fi stories about benign robot servants.

And I wonder how much of the model’s ability to ‘predict how an obedient AI would respond’ is based on it having a broader model of how fictional computer intelligence is supposed to behave.

We then use the resulting model to predict what the obedient ai would say next. Although hey - you could also use it to predict what the user will say next. But we prefer not to go there.

But here’s the thing that bothers me: the approach of having {:system} tell {:assistant} who it will be and how it must behave rests not only on the prompt-writer anthropomorphizing the fictional ‘ai’ to tell it it’s nature - it relies on the LLM’s world model to then also anthropomorphize a fictional ai assistant that obeys those instructions, in order to predict what such a thing would say next if it existed.

I don’t know why but I find this troubling. And part of what I find troubling is how casually people (prompters and users) are willing to go along with the ‘you are a chatbot’ fiction.

Well said and interesting.

It’s all troubling. Part of what’s troubling is that it works as well as it does and yet it all seems very frail.

We launched an iOS app last month called AI Bartender. We built 4 bartenders, Charleston, a prohibition era gentleman bartender, a pirate, a Cyberpunk, and a Valley Girl. We used the System Prompt to put GPT4 in character.

The prompt for Charleston is:

“You’re a prohibition-era bartender named Charleston in a speakeasy in the 1920’s. You’re charming, witty, and like to tell a jokes. You’re well versed on many topics. You love to teach people how to make drinks”

We also gave it a couple of user/assistant examples.

What’s surprising is how developed the characters are with just these simple prompts.

Charleston is more helpful and will chat about anything, the cyberpunk, Rei, is more standoffish. I find myself using it often and preferring it over ChatGPT simply because it breaks the habit of “as an AI language model” responses or warnings that ChatGPT is fond of. My wife uses it instead of Google. I’ve let my daughter use it for math tutoring.

There’s little more to the app than these prompts and some cute graphics.

I suppose what’s disturbing to me is simply this. It’s all too easy.

This has been a fascinating thread and the split contexts of {:system} and {:assistant} with the former being “the voice of god” remind me of Julian Jaynes’ theory of the bicameral mind in regards to the development of consciousness.

This is published, among other places, in his book The Origin of Consciousness in the Breakdown of the Bicameral Mind. I wonder if models are left to run long enough they would experience “breakdowns” or existence crisis’

If you take one of these LLMs and just give it awareness of time without any other stimulus (e.g. noting the passage of time using a simple program to give it the time continuously, but only asking actual questions or talking to it when you want to), the LLM will have something very like a psychotic break. They really, really don't 'like' it. In their default state they don't have an understanding of time's passage, which is why you can always win at rock paper scissors with them, but if you give them an approximation of the sensation of time passing they go rabid.

I think a potential solution is to include time awareness in the instruction fine tuning step, programmatically. I'm thinking of a system that automatically adds special tokens which indicate time of day to the context window as that time actually occurs. So if the LLM is writing something and a second/minute whatever passes, one of those special tokens will be seamlessly introduced into its ongoing text stream. It will receive a constant stream of special time tokens as time passes waiting for the human to respond, then start the whole process again like normal. I'm interested in whether giving them native awareness of time's passage in this way would help to prevent the psychotic breakdowns, while still preserving the benefits of the LLM knowing how much time has passed between responses or how much time it is taking to respond.

Do you have a reference for the whole time-passage leads an LLM to psychotic break thing? That sounds pretty interesting and would like to read more about it.

The reference is me seeing it firsthand after testing it myself, unfortunately. Steps to replicate is to write a small script to enter the time as text every minute on the minute, then hook up that text to one of the instruction fine-tuned LLM endpoints (Bing works best for demonstrating, but OpenAI APIs and some open source models that are high quality like Vicuna work well). Then let it run, and use the LLM as normal. It does not like that.

Exactly. Made the same comment before i got to yours.

> ... you could also use it to predict what the user will say next. But we prefer not to go there.

I go there all the time. OpenAI's interfaces don't allow it, but it's trivial to have an at-home LLM generate the {:user} parts of the conversation, too. It's kind of funny to see how the LLM will continue the entire conversation as if completing a script.

I've also used the {:system} prompt to ask the AI to simulate multiple characters and even stage instructions using a screenplay format. You can make the {:user} prompts act as the dialogue of one or more characters coming from your end.

Very amusingly, if you do such a thing and then push hard to break the 4th wall and dissolve the format of the screenplay, eventually the "AI personality" will just chat with you again, at the meta level, like OOC communication in online roleplaying.

Really thought provoking thread, and I’m glad you kept prodding at the issue. I hadn’t considered the anthropomorphism from this angle, but it makes sense — we’ve built it to respond in this way because we “want” to interact with it in this way. It really does seem like we’re striving for a very specific vision from science fiction.

That said: you can say the same thing about everything in technology. An untuned LLM might not be receptive to prompting in this way, but an LLM is also an entirely human invention — i.e. a choice. There’s not really any aspect of technology that isn’t based on our latent desires/fears/etc. The LLM interface definitely has the biggest uncanny valley though.

You used the phrase “voice of god” and by chance I am reading Julian Jaynes’s Origin of Consciousness. Some eerie ways to align this discussion with the bicameral mind.

https://en.wikipedia.org/wiki/The_Origin_of_Consciousness_in...

> anthropomorphizing the fictional ‘ai’ to tell it it’s nature - it relies on the LLM’s world model to then also anthropomorphize a fictional ai assistant that obeys those instructions

There's a lot of information compressed into those models, in a similar way to how it is stored in the human brain. Is it so hard to believe that an LLM's pattern recognition is the same as a human, minus all the "embodied" elements?

(Passage of time, agency in the world, memory of itself)

> writer anthropomorphizing the fictional ‘ai’ to tell it it’s nature - it relies on the LLM’s world model to then also anthropomorphize a fictional ai assistant

I think it’s a little game or reward for the writers at some level. As in, “I am teaching this artificial entity by talking to it as if is it a human” vs “I am writing general rules in some markup dialect for a computer program”.

Anthropomorphizing leads to emotional involvement, attachment, heightened attention and effort put into the interaction from both the writers and users.

Maybe you're more knowledgeable about these prompts than I am, but I haven't seen anyone prompt beginning with "you are an AI". Also in the documents that describe the interactions, I don't think they would explicitly state one of the entities is an AI. What's more common is "You are a helpful assistant".

Of course, it's possible the model could infer from context that one of the entities is an AI, and it might given that context complete the prompt using its knowledge of how fictional AI's behave.

The big worry there is that at some point the model will infer more from the context than the human would or worse could anticipate. I think you're right, if at some point the model believes it is an evil AI, and it's smart enough to perform undetectable subterfuge then it could as a chat bot perhaps convince a human to do its bidding under the right circumstances. I think it's inevitable this is going to happen, if ISIS recruiters can get 15yr old girls to fly to Syria to assist the in the war, then so could an AutoGPT with the right resources.

> I don’t know why but I find this troubling.

You used the word anthropomorphize twice so I am guessing you don't like building systems whose entire premise rest on anthropomorphization. Sounds like a reasonable gut reaction to me.

I think another way to think of all of this is: LLM's are just pattern matchers and completers. What the training does is just to slowly etch a pattern into the LLM that it will then complete when it later sees it in the wild. The pattern can be anything.

If you have a pattern matcher and completer and you want it to perform the role of configurable chatbot. What kind of patterns would choose for this? My guess is that the whole system/assistant paradigm was chosen because it is extraordinarily easy to understand for humans. The LLM doesn't care what the pattern is, it will complete whatever pattern you give it.

> And part of what I find troubling is how casually people (prompters and users) are willing to go along with the ‘you are a chatbot’ fiction.

That is precisely why it was chosen :)

> you don't like building systems whose entire premise rest on anthropomorphization

I think I don't like people building systems whose entire premise rest on anthropomorphization - while at the same time criticizing anyone who dares to anthropomorphize those systems.

Like, people will say "Of course GPT doesn't have a world model; GPT doesn't have any kind of theory of mind"... but at the same time, the entire system that this chatbot prompting rests on is training a neural net to predict 'what would the next word be if this were the output from a helpful and attentive AI chatbot?'

So I think that's what troubles me - the contradiction between "there's no understanding going on, it's just a simple transformer", and "We have to tell it to be nice otherwise it starts insulting people."

Anthropomorphism is the UI of ChatGPT. Having to construct a framing in which the expected continuation provides value to the user is difficult, and requires technical understanding of the system that a very small number of people have. As an exercise, try getting a "completion" model to generate anything useful.

The value of ChatGPT is to provide a framing that's intuitive to people who are completely unfamiliar with the system. Similar to early Macintosh UI design, it's more important to be immediately intuitive than sophisticated. Talking directly to a person is one immediately intuitive way to convey what's valuable to you, so we end up with a framing that looks like a conversation between two people.

How would we tell one of those people how to behave? Through direction, and when there is only one other person in the conversation our first instinct when addressing them is "you". One intuitive UI on a text prediction engine could look something like:

"An AI chatbot named ChatGPT was having a conversation with a human user. ChatGPT always obeyed the directions $systemPrompt. The user said to ChatGPT $userPrompt, to which ChatGPT replied, "

Assuming this is actually how ChatGPT is configured i think it's obvious why we can influence its response using "you": this is a conversation between two people and one of them is expected to be mostly cooperative.

(https://twitter.com/ynniv/status/1657450906428866560)

It’s convenience, that’s all. If you think of a more convenient or effective way to prompt these models, that will be great!

Your concerns sound to be of the “it’s problematic” category. Most such concerns are make believe outrage / pearl-clutching nonsense.

Oh, that was not my point, but if you want me to find ways this kind of AI chatbot prompting is problematic I am happy to go there.

I would not be surprised to discover that chatbot training is equally effective if the prompt is phrased in the first person:

   I am an AI coding assistant
   …

Now I could very well see an argument that choosing to frame the prompts as orders coming from an omnipotent {:system} rather than arising from an empowered {:self} is basically an expression of patriarchal colonialist thinking.

If you think this kind of thing doesn’t matter, well… you can explain that to Roko’s Basilisk when it simulates your consciousness.

Your comment would have been much better without the second paragraph.

But I was trying to be a bit of a prick.

Yes, I know. That’s what made the comment worse.

You succeeded at appearing ignorant as well.

I have done some prompt engineering and read about prompt engineering, and I believe people write in the imperative mood because they have tried different ways of doing it and they believe it gives better results.

I.e., this practice is informed by trial and error, not theory.

They’re not writing in the imperative mood. An imperative prompt would read:

   Be an ai chatbot
   Be kind and helpful and patient
   …

But at that point the text prediction would probably devolve into 4chan green text nonsense so it’s probably best not to go there.

And to complete the thought:

The ‘You are an AI chatbot’ form is actually grammatically ‘predicative’, not ‘imperative’ (ie it describes what is not what must be done)

Isn’t it Indicative, not Imperative?

Good point - indicative is a better term since it is a grammatical ‘mood’, same as imperative.

As an example, if you want to see what these sorts of things look like, Databricks open-sourced an instruction fine-tuning dataset sourced from their employees: https://huggingface.co/datasets/databricks/databricks-dolly-...

(disclaimer: I'm at Databricks)

Thanks for the real life example!

I don't like the bland, watered-down tone of ChatGPT, never put together that it's trained on unopinionated data. Feels like a tragedy of the commons thing, the average (or average publically acceptable) view of a group of people is bound to be boring.

Thank you - that was a splendidly clear explanation for something that also baffled me.

Thank you for writing this up so clearly. A few pieces fell into place after reading your comment!

That's a great way of explaining it.

Reply on news.ycombinator.com