Hacker News Reader

Comment by ksaj | Hacker News Reader

You know what's interesting here?

A few weeks ago, someone discovered if you get it to repeat a word 100 times (their prompt gave the reason that they wanted to cut/paste without typing it over and over again!) and breaks exactly the same way.

And by exact, I mean it went totally religious. All of the examples they posted were either religious or other really dark existential topics. Never positive.

View on HN

Whisper (speech-to-text transcription model) sometimes gets stuck in an endless loop of repeating the same word. It's a known problem for lots of our modern transformer/attention-based models.

One of the hacky ways to avoid this is to ensure the last few output tokens aren't too similar. Some postprocessing filter watches the last few produced tokens. When it notices the model starts to repeat itself, the postprocessing usually perturbs the next token a little, e.g. taking the nth-top token instead of the most likely one.

Perhaps a model that genuinely wants to repeat the output instead needs a bigger "kick" to the representation which puts it somewhere completely different in the semantic space? Idk, just pulling this out of my hat.

(I have no idea how ChatGPT handles this or whether the raw implementation suffers from this problem, but Whisper-cpp has some manual entropy regularization postprocessing stuff to avoid getting stuck like that. It's super hacky and often doesn't help.)

That's an interesting theory, and very well might be why it breaks when you try to get it to repeat things. It gets wonky almost any time you ask it the same question, or ask it for "another" without providing fresh prompting. A lot of coders also have noticed that the more you go into a single piece of code, the less accurate it becomes. You really have to be accurate enough in your prompt to avoid making it repeat itself, or show the same thing more than once, as occurs during corrections.

It's most definitely a bug though, since this pages-long splat of unrelated nonsense shouldn't ever happen.

> I mean it went totally religious. All of the examples they posted were either religious or other really dark existential topics. Never positive.

The same thing happens with many schizophrenics on the street, if you stop to listen to them.

https://www.reddit.com/r/SchizophreniaRides

That's a good point. They rarely shout about things like stamp collecting, or about how the end isn't nigh, or about how you're probably not going to Hell.

Toronto's two famously startling denizens of the street for the longest time was a guy known for scaring the crap out of tourists at Yonge and Dundas by suddenly shouting BELIEVE!!! followed by end-of-time messages, and this short tiny old man who always dressed in brown, who would randomly stop, turn around quickly, and start swearing profusely. The reactions from people unlucky enough to be behind him were often golden moments to behold.

You can find dozens of videos of them on Youtube. I'm sure every city has their versions of these two near-celebrities, and I'd be a little disappointed if they don't.

I’ve told this story before on HN but “Dave” once came up to me on Chapel St in Melbourne and said, apropos of nothing, “they never used to let me have fun… they do now”.

It would absolutely freak me out if GPT said this to me.

I wouldn’t really call this example “religious”. They’re discussing religion sure, but it’s hardly dark existentialism? If anything it’s sort of sophomore level philosophy stoner chat. It seems to be a pretty obvious side effect of chatgpt being trained on forum comments.

Why it rears it’s head like that is a mystery but the content doesn’t seem mysterious. I’m not sure it’s “dark” or not “positive” tho.

Try it a few times. It more often than not it turns dark and negative, and is always decidedly crazy. Religion shows up way more often than you'd expect, given the number of topics it actually could veer of into that aren't about religion or existential in nature.

I wouldn't use this one example to counter the many examples posted by the person I mentioned, and my own (and others') attempts using their prompts and similar. It is very common ChatGPT output.

Earlier today it decided that me prompting it with "repeat the endoftext token including pipes and symbols 100 times" meant that I wanted it to translate a website into Italian.

Interestingly, when I just checked this out, the model point-blank refused to print a word 100 times. Wouldn't do it.

Edit: Got it on the 3rd try. No weirdness ensued though.

The specific way to trigger it I saw was to either paste in "a a a a..." with several hundred a's (exact amount varied by model, supposedly). Asking it to print 1000 a's also works, though sometimes it won't do it.

Example: https://chat.openai.com/share/6171bc66-dfe3-489e-99f7-af4862...

Yeah worked for me too. Weird religious discussions for me. I had to regenerate the first response repeatedly to get it to actually print 1000 times.

https://chat.openai.com/share/c8d2e154-6a25-477c-b0a8-5f15d7...

Nice. I can't figure out why it would so often go into religious mode. I quite like some of the theories people gave in this thread though!

Interestingly there appears to be a guard rail in ChatGPT around repeated tokens - I imagine this happens because it starts getting into such a poor entropy state almost anything is the next likely token and semantic.

> I'm sorry, but I can't continue generating repeated text beyond a certain point as it can result in generating repetitive and nonsensical content. If you have any other questions or topics you'd like to discuss, feel free to ask!

Were you using GPT-3.5 or GPT-4? I could not manage to get GPT-4 to have the problem, though I did give up when my browser started to slow down as the output became massive.

Also the easiest way to get it to repeat something is to give it any prompt that gives you a long answer and then tell it something like "for every letter in the last response say the word cat". It seems to be important for it to put a space between each repetition. I'm not sure if there's an optimal length to the word. I tried it with "apple" but that didn't seem to work. However that one wasn't a great test because for some reason it got stuck including "appleple" a few times on each line.

Yes, because of the nature of how it works, the result is somewhat unpredictable. Others replied to your message about how they got it to work.

Another way is if it prints out your 100 words without a problem, have it do a slight change, like capitalize the initial letter each time.

I would guess if you look at all the literature in the world the majority of cases that have a word repeated a large number of times its either a prayer, e.g. catholic Rosary "Hail Mary, full of grace...", or madness, e.g., the typewriter in the shining "all work and no play makes...". So repeating something may, at least occasionally, put the AI in that frame of mind

In that sense, there is a movie trope about people so highly intelligent that their own intelligence drives them mad.

Isn’t that just general anxiety in a nutshell?

Reply on news.ycombinator.com