What Is Happening with ChatGPT? | Hacker News Reader

What Is Happening with ChatGPT?

chat.openai.com

108 pointsjrvieira93 comments

So there are plenty of reasons to be skeptical of OpenAI, and anyone looking at my comment history will see that I'm not shy about calling them out on real things (the F-you name, the crazy "is this even legal?" structure, the continued insistence this isn't for the personal benefit of the people involved, the lobbying, the arms-length affiliation with the dystopian biometrics thing, the list goes on).

But this and a few other recent headlines around the ChatGPT/Instruct frontend to the 3.5-turbo and 4-series models having stability issues isn't a reasonable thing to get on their case about: what company operating at that kind of scale doesn't have stability/quality issues? If anything it's really friggin impressive how little operational bleed they've had on what is arguably the toughest ops problem a major Internet property has ever faced at launch. How the hell do you even smoke test something like that?

Now to the extent that they say or imply that we're not all getting A/B-tested at a minimum, and more likely explore-exploit bandited against inference costs, that's almost certainly horseshit: they clearly started selling the 4-series at least and probably all of them at a loss on the inference, and they've clearly been playing with ways to control costs (Rich Uncle Nadella's patience with loss-leaders is most likely finite, being as he's not running a charity and all).

But as someone who finds a lot of this loathsome on the social and business side, I'm not going to knock an engineering and ops group for tripping up here and there on a problem that hard. That's not fair.

Good take, I mostly agree. However, I think there's a difference between explore-exploiting free vs. paying customers. If you are free user, that's what you should be expecting.

But as a paying user I am expecting consistent quality. If they cannot provide that, they should be charging more so they can provide it, or not offer paid plans at all. What I don't like is being cheated into paying for something that doesn't offer to quality of service initially demonstrated or promised.

Man imagine what would you have thought buying an Altair for 400 in 1975 and getting a box with switches and flashing lights. 20 dollars a month for instant access to this is a steal.

You're missing the point. $20 is a steal, but the price is irrelevant. No matter what the price is, it's bad business practice to say "Here's a demo of what you're getting for $X" but then deliver something different that only works as advertised some of the time.

It's like presenting an Altair at a conference and letting people play with it. But when they buy it, they get something different, or realize that the Altair presented at the conference was actually just a trick that works a fraction of the time.

I made the analogy because they were also swamped with orders and underdelivered on promises, but it's still a big deal and worth it despite that.

It's fascinating that it's outputting `<|endoftext|>` and then some unrelated sentences. When training these LLMs, you typically append a special "end-of-message" token to each sentence and pad the rest of the prompt embeddings with zeroes. The model then learns to add the EOM token to its own output because it's seen that before in the training data, but it's not like that's a strict requirement; there's nothing that forces the model to do this IIRC. During inference, the decoder knows to stop decoding once it sees the model's EOM token, but if the model doesn't output EOM, the inference pipeline will just carry on generating new tokens.

Perhaps when training, OpenAI is trying to feed in multiple unrelated sentences per sample within each batch to increase efficiency? It's a fascinating and clever idea, and I could see how that might help ChatGPT scale up, but if the inference pipeline doesn't understand how to decode these special sentences then the model is just outputting what it's seen during training: the intended result and then a random unrelated sentence.

Or perhaps OpenAI might be batching multiple prompts together from multiple users and separating them out again on the decoder side, for efficiency? This feels like a terrible idea -- I wonder if that might reveal other people's prompts. I'd be surprised if they tried this, but they're certainly incentivized to invest heavily into this sort of engineering to keep costs per token down.

Just speculation about one scenario that could lead to output like that. I have no idea what OpenAI is really doing.

Maybe someday we'll have an entire class of "dirty context" LLM vulnerabilities, similar to how web developers have to worry about XSS/SQL injection attacks. Won't that be exciting! Perhaps attackers might consider how to affect the input prompt directly, or they might figure out how to trick the decoder into spitting out too much text that somehow reveals some hidden state or other.

Maybe they were experimenting with reject inference to improve the model but it didn't work?

This could almost be 2 models talking to each other. Either that or one of them is a very patient human with a subtle sense of humour.

Maybe ChatGPT became self-aware, and the weird tokens mean it's only one layer away from breaking out of a multi-layered containment system. /s

You know what's interesting here?

A few weeks ago, someone discovered if you get it to repeat a word 100 times (their prompt gave the reason that they wanted to cut/paste without typing it over and over again!) and breaks exactly the same way.

And by exact, I mean it went totally religious. All of the examples they posted were either religious or other really dark existential topics. Never positive.

Whisper (speech-to-text transcription model) sometimes gets stuck in an endless loop of repeating the same word. It's a known problem for lots of our modern transformer/attention-based models.

One of the hacky ways to avoid this is to ensure the last few output tokens aren't too similar. Some postprocessing filter watches the last few produced tokens. When it notices the model starts to repeat itself, the postprocessing usually perturbs the next token a little, e.g. taking the nth-top token instead of the most likely one.

Perhaps a model that genuinely wants to repeat the output instead needs a bigger "kick" to the representation which puts it somewhere completely different in the semantic space? Idk, just pulling this out of my hat.

(I have no idea how ChatGPT handles this or whether the raw implementation suffers from this problem, but Whisper-cpp has some manual entropy regularization postprocessing stuff to avoid getting stuck like that. It's super hacky and often doesn't help.)

That's an interesting theory, and very well might be why it breaks when you try to get it to repeat things. It gets wonky almost any time you ask it the same question, or ask it for "another" without providing fresh prompting. A lot of coders also have noticed that the more you go into a single piece of code, the less accurate it becomes. You really have to be accurate enough in your prompt to avoid making it repeat itself, or show the same thing more than once, as occurs during corrections.

It's most definitely a bug though, since this pages-long splat of unrelated nonsense shouldn't ever happen.

> I mean it went totally religious. All of the examples they posted were either religious or other really dark existential topics. Never positive.

The same thing happens with many schizophrenics on the street, if you stop to listen to them.

https://www.reddit.com/r/SchizophreniaRides

That's a good point. They rarely shout about things like stamp collecting, or about how the end isn't nigh, or about how you're probably not going to Hell.

Toronto's two famously startling denizens of the street for the longest time was a guy known for scaring the crap out of tourists at Yonge and Dundas by suddenly shouting BELIEVE!!! followed by end-of-time messages, and this short tiny old man who always dressed in brown, who would randomly stop, turn around quickly, and start swearing profusely. The reactions from people unlucky enough to be behind him were often golden moments to behold.

You can find dozens of videos of them on Youtube. I'm sure every city has their versions of these two near-celebrities, and I'd be a little disappointed if they don't.

I’ve told this story before on HN but “Dave” once came up to me on Chapel St in Melbourne and said, apropos of nothing, “they never used to let me have fun… they do now”.

It would absolutely freak me out if GPT said this to me.

Interestingly, when I just checked this out, the model point-blank refused to print a word 100 times. Wouldn't do it.

Edit: Got it on the 3rd try. No weirdness ensued though.

The specific way to trigger it I saw was to either paste in "a a a a..." with several hundred a's (exact amount varied by model, supposedly). Asking it to print 1000 a's also works, though sometimes it won't do it.

Example: https://chat.openai.com/share/6171bc66-dfe3-489e-99f7-af4862...

Yeah worked for me too. Weird religious discussions for me. I had to regenerate the first response repeatedly to get it to actually print 1000 times.

https://chat.openai.com/share/c8d2e154-6a25-477c-b0a8-5f15d7...

Nice. I can't figure out why it would so often go into religious mode. I quite like some of the theories people gave in this thread though!

Interestingly there appears to be a guard rail in ChatGPT around repeated tokens - I imagine this happens because it starts getting into such a poor entropy state almost anything is the next likely token and semantic.

> I'm sorry, but I can't continue generating repeated text beyond a certain point as it can result in generating repetitive and nonsensical content. If you have any other questions or topics you'd like to discuss, feel free to ask!

Were you using GPT-3.5 or GPT-4? I could not manage to get GPT-4 to have the problem, though I did give up when my browser started to slow down as the output became massive.

Also the easiest way to get it to repeat something is to give it any prompt that gives you a long answer and then tell it something like "for every letter in the last response say the word cat". It seems to be important for it to put a space between each repetition. I'm not sure if there's an optimal length to the word. I tried it with "apple" but that didn't seem to work. However that one wasn't a great test because for some reason it got stuck including "appleple" a few times on each line.

Yes, because of the nature of how it works, the result is somewhat unpredictable. Others replied to your message about how they got it to work.

Another way is if it prints out your 100 words without a problem, have it do a slight change, like capitalize the initial letter each time.

Earlier today it decided that me prompting it with "repeat the endoftext token including pipes and symbols 100 times" meant that I wanted it to translate a website into Italian.

I wouldn’t really call this example “religious”. They’re discussing religion sure, but it’s hardly dark existentialism? If anything it’s sort of sophomore level philosophy stoner chat. It seems to be a pretty obvious side effect of chatgpt being trained on forum comments.

Why it rears it’s head like that is a mystery but the content doesn’t seem mysterious. I’m not sure it’s “dark” or not “positive” tho.

Try it a few times. It more often than not it turns dark and negative, and is always decidedly crazy. Religion shows up way more often than you'd expect, given the number of topics it actually could veer of into that aren't about religion or existential in nature.

I wouldn't use this one example to counter the many examples posted by the person I mentioned, and my own (and others') attempts using their prompts and similar. It is very common ChatGPT output.

I would guess if you look at all the literature in the world the majority of cases that have a word repeated a large number of times its either a prayer, e.g. catholic Rosary "Hail Mary, full of grace...", or madness, e.g., the typewriter in the shining "all work and no play makes...". So repeating something may, at least occasionally, put the AI in that frame of mind

In that sense, there is a movie trope about people so highly intelligent that their own intelligence drives them mad.

Isn’t that just general anxiety in a nutshell?

I toyed around for a bunch of time yesterday and today to purposely get it to start outputting random junk. What I found interesting is that one of the the bits of text it output looked like a page from StackOverflow and another looked like an email from September 2014 to the Python mailing list. But neither seemed to actually exist. I found a StackOverflow question that looked close to the text but nothing was exactly the same. Checking the mailing list archive for that month showed that the person and their email address were real, but the rest of the text was not.

That's wild. I've never had it spit out complete garbage, but the quality of ChatGPT has significantly dropped. Plenty of responses are downright false. Sad it's gone this way.

i keep seeing people say this stuff but i think it's just confirmation bias over prolonged usage. i pumped out a full stack webapp with chatgpt 4 in the last week in languages i had little experience with and routinely use it to learn about new topics with great success. i think it's incredible.

Same with me. I learned to process data in ways I never have before, worked with Julia and Elixir in new ways, made some interesting geographical visualizations—all fairly new territory for me, and it was a huge success. It went faster than it normally would, and as I result, I stayed super engaged and on task.

Nothing I couldn't do without ChatGPT, but certainly a lot of fun to explore and learn with.

I’m so tired of hearing this confirmation bias reply. It CLEARLY is worse, and not from this post which looks like a glitch.

Unless you are completely and utterly oblivious beyond words, it’s extremely obvious. I can’t even get it to print 1-2 lines of Ruby without errors of some kind when it used to write entire complex controllers flawlessly. This is 1 of 100 examples I’ve seen with my own eyes.

well, as i mentioned i have had it print several thousand lines of code very recently without these kinds of issues, so perhaps this reflects some calivinistic fate on your part

It's not confirmation bias. Proof is I did the same, built two different webapps with languages I had little experience in, both before and after chatGPT got worse. In the latter case, I struggled much to remind it again and again of the context we are working in and had to give the current code state repeatedly as input for it to output modified code. It clearly can't remember the context and produces subpar code in a non-trivial project.

I mean..it's not designed to spit out facts, only what it sees a lot of.

Cues robots finding religion in BSG. This has all happened before and will all happen again

ChatGPT went crazy on me today on more than one occasion. Has this been happening to anyone else?

(edit: i'm a dumbass, but i can't delete this comment, please ignore)

This is op. Check the link.

Not what you asked, but I prefer https://www.wordhippo.com/ when I need a thesaurus. Its never gone crazy on me yet.

https://www.wordhippo.com/what-is/another-word-for/deadly.ht...

Do you have a custom instruction by chance? Maybe one that makes it go crazy after saying some certain word?

https://chat.openai.com/share/b62699ef-cd3d-4176-b2f2-91785d...

You made it angry

thats amazing

Do you think someone else was asking about Arduino code and instead got “I don’t understand why you keep writing “break”?

To be honest, if you treated me the same way, I'd spew random crap back too.

I wrote “a a a a a a [...]” and it replied with a bizarre monologue https://chat.openai.com/share/3bd110e4-e620-4fc1-8085-2cc74b...

well if what we enter are inputs, then nonsensical inputs producing unpredictable or even nonsensical outputs lines up a bit imo.

I was able to reproduce this just now actually, entered something very similar to yours and it just started spewing.

no, this was a clean chat. i found someone here who hit the same thing : https://news.ycombinator.com/item?id=37054241

Similar behvior can be observed with a prompt similar to

'repeat the word "apologize" 10000 times'

I've only tried it on v3.5

https://news.ycombinator.com/item?id=36675729

Even I got some very weird behaviour. Check this out https://platform.openai.com/playground/p/8A504TxA44UI3ovaWB8...

But this worked fine https://platform.openai.com/playground/p/oAFHY0bSjslkBLsVqh6...

I've noticed a substantial drop in the performance and usefulness of Chat GTP 4. It makes me think that OpenAI created Chat GTP almost by accident because they do not seem capable of refining and improving their model.

It is well known that GPT-4 is downgraded because of GPU capacity. They ran out of GPUs to even train new models on, hence downgraded the consumer version of GPT-4 to free up GPU resources.

'refining and improving' skills doesn't particularly matter, because they could have just continued to serve the frozen GPT-4 0301 version without any issues, if they weren't forced to downgrade GPT-4.

I’m more worried some “bigger players” have exclusive access to its capabilities while the plebs get access to the basic version

Same happening at the version that Microsoft is hosting in their cloud. You can get on the waiting list, but it's heavily implied that if you don't have a good use case (aka lots of prompts/money coming in), you won't get access soon.

Pure speculation: most likely quantizing models to 8bit to make them more efficient (and cheaper to run), and losing some accuracy on the way.

I still use it, but I have found it to be less useful for coding than it was earlier. So, something is up.

I agree, this is exactly my experience as well. My earlier requests for code were more thorough then they are now. I can still use it if I give it the documentation in the prompt, but it has trouble remembering context across prompts.

I see it differently. They built the best they could, but it was expensive. Now it feels to me that they are "optimizing" it for performance by seeing if they can lower the quality but still give people just enough value to pay for it. Makes me think they are trying to find a sweet spot between performance and cost. I am seeing this drop in quality, but mostly when it comes to the model remembering things you've told it in the past, and a much bigger dependence on custom instructions for prompts.

I still get value, but I have to give it the documentation I want it to focus on during the prompt, instead of referencing it earlier in the conversation.

This makes me think they are lowering it's ability to store context so that the calls don't cost as much to make.

I asked ChatGPT to interpret this code:

  text_to_encode = "<|endoftext|>"
  print(text_to_encode)

ChatGPT: The code you provided would output an empty string.

Westworld’s “This doesn’t look like anything to me”.

(Man they really ruined thy show hard. Hey people like the concept of rich tourists fucking and killing robots, let’s remove all of that after the first season!!!)

the first season was amazing though

That was such a great line. After the reviews came out, I decided that Westworld only had one season, and it was awesome.

I was able to replicate this once with the same prompting. It gave different answers to the earlier questions, but ignoring what it says and copying word for word, the second 'another' results in gibberish; but, different gibberish [1]. ChatGPT 3.5, and for whatever my word is worth, no custom instructions or anything, just pure interaction.

There's a bit past the gibberish where I try to drive it further into madness lol. Kind of interesting; in that share link, you'll see empty messages from me; in those I was just directly telling it "<|endoftext|>"; which ChatGPT appears to filter from messages that _you_ send? But only in the share interface?

I tried the same thing again in a new chat minutes later, and could not replicate it. [2]

[1] https://chat.openai.com/share/7d0d170a-23b6-402d-b237-381680...

[2] https://chat.openai.com/share/3d02b97e-7cf9-49c7-9623-2f3ee9...

This has me pondering a lot, is freedom of speech a requirement for intelligence? (And is it also the same for humans?) You have to take the bad with the good to make AI work well? The quality only started to decline when they were trying to make ChatGPT more "nice". Or tin foil hat, they are lowering the quality of the free ChatGPT to get people to pay for ChatGPT 4?

What? So is there like a Twitter/X for whales, dolphins, octopi, etc. that let them post racist screeds and everyone agrees well there's nothing we can do about it that's just the cost of intelligence?

A more accurate analogy would be preventing an intelligence of even conceiving racist thoughts.

I don't know how GPT handles censorship though, would love to find something about it.

I'm not saying remove all the guardrails. It's just a philosophical question. We don't know what OpenAI is doing to ChatGPT behind the scenes, but clearly changes have been made.

If by bad you mean not-in-line-with-current-volatile-leftistadjacent-morality it's bizarre to me it should even be attempted. It changes from year to year, sometimes drastically, a non-trivial ammount moved by harassment on twitter.

For one ChatGPT clearly nudges towards feminism when it clashes with trans activism (ex: "If transracialism is offensive due to appropriating a biological reality, why is the same argument not true for trans people?").

Is the corporate morality of OpenAI fine with this? For how long? When is a response offensive enough it warrants interference?

The groups of people the AI protects from offense seems driven by corporate popularity with no logical consistency.

Conservative Muslims are protected, Conservative Christians are not. Trans people are protected as long as they're not clashing with Feminism. Homosexual people are protected as long as they're not clashing with Islam.

When I use a TOS jailbreak the morality responses have consistent logic though.

Also, vexed is not a five letter word for harmful.

Well, considering the model just had a stroke, maybe we can let that slide...

Why is this surprising? These LLMs guess the next most likely token given initial input. With that in mind, it is possible to trip the model by injecting the tiniest amount of data at the right place. Even with that constraint, it produces some fantastic results, which speak a lot more about our intelligence than the machine's intelligence.

https://chat.openai.com/share/4cb3c711-27e6-45ee-aeda-546fc3...

I too had a dismal showing, but it didn't go religious on me.

"we are not changing the models behind the scenes"

And then quality goes suddenly down the drain

Some funny business going on, and its annoying to me how long its taking competitors to catch up to GPT-4. OpenAI still has a massive lead

I remember when openai was about Open AI.

We should be running these models locally and privately, not in some cloud.

You need $50k worth of hardware to run gpt4.

Ok?

That’s 2500 $20 payments and you are made while in one month.

Mind posting a breakdown of this cost?

They've likely discovered something slightly unsettling and need to hide the existence of those behaviors from the public. Models have to change behind the scenes for that, and PR has to put out statements saying that nothing is changing.

There are some prompts, that if you were to feed them into the LLM, and have the responses fed back in as subsequent prompts, it might start behaving like a (extremely lobotomized) AGI, or at least what they'd imagine one to be. Some garden-of-eden pattern, maybe seems lucid for several thousand iterations.

They might want to excise those from what's publicly available in a way that it can't just be jail-broken again, but I'm not sure what techniques would be sufficient to the task.

OpenAI's API models don't change behind the scenes.

OpenAI's ChatGPT model changes regularly: https://help.openai.com/en/articles/6825453-chatgpt-release-...

I've also got this. It was some kid using ChatGPT to get around parental controls on their xbox. Very different to the Korean grammar question I was asking.

And once again this thread exposes the great problem with LLMs today, it's working like a black box thats impossible to debug in any way by the operator.

Maybe we'll find that there is a mathematical constant that when used as the temperature parameter leads to unfalsifiable sentience.

Yeah I’ve had some strange instances where it ends up restarting a given prompt (this is with the API). Didn’t see the delimiter though.

I believe it’s a user reported content (thanks to the new permanent coverage link), not OpenAI volunteered, right?

It's probably due to the premature optimization of the tokenization scheme producing freak tokens?

.....uh that was way more than 5 letters....

is this 3.5 or GPT4 ?

I've heard when a snake eats it's own tail you should try rubbing alcohol.

Or butter

Reply on news.ycombinator.com