Hacker News Reader

Comment by numpad0 | Hacker News Reader

Isn't it fascinating that despite `while true; do claude --yolo` over a weekend being all it takes to port some project across platforms, LLMs completely fall apart when it comes to speaking grammatical and natural Japanese?

Free tier Gemini CLI literally writes Android app for me by just endlessly wondering in English. AGI's here. And it struggles with Japanese. How!?

View on HN

The simple truth is that LLMs are still bullshitting their way through life.

In some fields and languages, this is easier, but as soon as you need the LLM to follow rigorous instructions, it'll fail.

A lot of languages require a lot more context to speak and understand than others. English is a very low-context language, at least in formal registers - it just gets very wordy and very specific, and it helps that it is very analytic and isolating, with a massive vocabulary.

But for e.g. Spanish, if a bare subjunctive verb (making the verb "<x>" something like "could <x>," "should <x>," or "would <x>" without specifying) is used in a sentence, there's no way to know from that sentence who or what that subjunctive is being applied to. The unwritten rule is: 1) if it's obvious who or what you've been talking about, then it applies to that person or thing, 2) if there are two or more things it could be about, then you should figure out a way to add more information to specify which, and finally 3) if there are no good candidates, the person is referring to themselves.

I've heard that there's a lot more of that in Japanese, a language that I don't think really has personal pronouns at all. For example, iirc when you refer to an emotional state and don't specify who you're talking about, it's automatically assumed that you're talking about yourself. I'm not too familiar with Japanese, but languages are simply different from each other, they're not substitution ciphers.

LLMs have trouble staying aware of the context as things go on even for a fairly short time, and in some languages, the meaning of the conversation infects every utterance. Seems a little similar to how LLMs are bad at lifetimes in Rust. I'm sure they'll gradually get better.

> struggles with Japanese

It doesn't mention mistranslating, so it's difficult to know the root of the problem is AI "struggling".

> It doesn't follow our translation guidelines. > It doesn't respect current localization for Japanese users, so they were lost.

I believe this is the root of the problem. There are define processes and guidelines, and LLM isn't following it. Whether these guidelines were prompted or not is unclear but regardless it should've been verified by the community leaders before it's GA'ed

That's not the root of the problem. The root of the problem is that LLMs just can't constitute a punctually Japanese understanding of text like that guideline and speak in Japanese with native fluency no matter what. I just know this from knowing both sides of English-Japanese language pair. And I find that somewhat fascinating in a sense.

Wow. That's crazy. I would be very interested in a blog post on this subject if you ever wrote one. I wonder how this affects the perception of LLMs in Japan

I'm surprised by that. I use ChatGPT for communication with Japanese clients and I was mistaken for a native speaker more than once. I make almost no corrections other than changing the punctuation from western to fullwidth, although to be fair it's mostly simple and technical language.

I took this to mean it's not translating things consistently. Like, a button in Firefox might say "Show all downloads" or "Open previous windows and tabs". These were localized a certain way, but an AI has no ability to check that. It will just translate them anew, which might be the same or it might translate it to something synonymous, but which then confuses users searching for the "Display all downloads" button or whatever.

To be fair, in my experience LLM-s struggle to some extent with nearly every language that isn't English. Compared to many others, English is a very simple language and it has the largest body of material to learn from to boot. The more nuanced a language, the more LLM-s spout garbage.

They do pretty good Russian, even to the point of very elaborate speech styles etc.

Russian-speaking web was pretty strong once. There was a lot of educated people in ex-USSR who were apt with tech, knew Russian well and were willing to use it.

That's very much in the past now. But it'll linger in the training data for a while.

It's funny because I'm pretty sure modern LLMs wouldn't exist if Alexandra Elbakyan didn't come up with the concept of a shadow library.

Reply on news.ycombinator.com