Hacker News Reader

Comment by Kuinox | Hacker News Reader

ChatGPT is a single pass for each character. It can't "think" in English then translate it. It will often respond in English instead of the language you are using. Yes, it's better in English than any other languages, because the English dataset is bigger, and that's why an English AI will always be slightly better with the same technology.

View on HN

That’s indeed its interface, but there is still a huge training phase during which the model was taught using mostly English, and it explains why some other languages interactions feel like they are some direct translation of English expressions (not to mention higher ways of thinking). Most of the thinking indeed is done during training, and then encoded in the weights which act like a huge cache of thoughts, but there is still a lot of learning on how to use the context when you emit this character (token), the attention layers, which is also a form of thinking that uses whatever new input you are giving to the model and applies a series of transformations (granted, finite in current models) to it.

But maybe the dataset for said language is just too small to produce correct content at all, and that's why ChatGPT is bad, not because "ChatGPT is thinking in English". Except when it default to answering in English, ChatGPT sounds like french when I talk to it in french.

The bigger the dataset, the best is it's capabilities, training it exclusively with a language don't mean it will be better at this language.

> ChatGPT is a single pass for each character.

Not for each token?

Yeah, each token, but that's not my point.

ChatGPT does not think and we’re anthropomorphizing it. Having said that, it does “sound” like they are thinking in en-US or as if they are machine-translating English boilerplates.

e.g. wouldn’t say “cake may be cutted by a knife” and instead would say “knives can cut cakes”, always. In languages which the first example would be more correct and the latter would be borderline incorrect, GPTs tend to prioritize the second pattern. And this isn’t ideal.

Reply on news.ycombinator.com