Hacker News Reader

Your japanese phrases are quite easy for machine translation, as they have clear subjects stated and not much ambiguity.

The main problem is the implicit subject due to omission of pronoms in regular japanese, among other ambiguous things that can only be correctly translated in context. Also ommision of gender, plural, etc. The huge decoder-only models so far do a much better job grasping/guessing that and staying consistent compared to small encoder-decoder MTL models.

This simple phrase any beginner in japanese would understand: "山田さん、猫いますか？はい、います。" "Yamada-san, neko imasuka? Hai, imasu."

    Google Translate now: Yamada-san, do you have a cat? Yes, I'm here.

    When tested a few years ago: “Mr Yamada, does he have a cat? Yes, it has a cat”

    LLama-3-8b-instruct: "Yamada-san, is there a cat? Ah, yes, there is."

    GPT 3.5: "Does Yamada-san have a cat? Yes, he/she does."

    gemini-1.5-pro-api-0409-preview: Does Mr./Ms. Yamada have a cat? Yes, they do.

    LLama-3-70B-instruct: "Yamada-san, do you have a cat? Yes, I do."

    GPT 4, Command-R+: "Mr. Yamada, do you have a cat? Yes, I do."

The two google translate ones are simply terrible. Technically, they are possible translations, but I would rate them as wrong.

All LLMs gave possible translations, but the larger the LLM, more likely to be correct the translation became. Well, the top two ones chose a gender, while the -san thing is a clever way to avoid chosing a gender, but not a very pure translation. If I had to give only one translation, I would pick either of those.

Also of note, while some even explained each element of their translation, none said that other possible translations were possible.

GPT4: 時々、私のために英語で説明してもらったり、彼のために日本語で説明してもらったり、その逆もしてもらいます。同じ文脈で言語を切り替えるのは全く問題ありません。英語と日本語の両方で説明が難しい様々なフレーズも、ChatGPTはサポートしてくれます。

GPT4->GPT4: Sometimes, I have explanations given in English for me, or in Japanese for him, and vice versa. Switching languages in the same context is no problem at all. ChatGPT also supports me with various phrases that are difficult to explain in both English and Japanese.

DeepL: 時々、私は英語で、彼は日本語で何かを説明する。同じ文脈で言語を切り替えても問題ありません。英語でも日本語でも説明しにくいフレーズもChatGPTは問題なく教えてくれます。

DeepL->DeepL: Sometimes I explain something in English and he explains something in Japanese. Switching languages in the same context is no problem. ChatGPT has no problem teaching me phrases that are difficult to explain in either English or Japanese.

First of all are you seriously using GPT 3 and not 4?

Second I’m giving you feedback from an actual Japanese person who used Google Translate all the time to go back and forth. Now using GPT 4 they say it is no contest way better than Google Translate. To a point where they can use it to understand and communicate professionally with English and Japanese business customers.

But you can even see it yourself taking actual articles from Japanese websites and running them through Google Translate versus GPT 4, GPT translates into much much more fluent English, same with the translation to Japanese.

Well, I'm actually an actual Japanese person too. As for using GPT 3.5, I just never paid for GPTs, sorry for that. The deltas just look marginal to me, I know everything on the Internet(/s) and I can read/write basic English, and I'm a bit of an crazy paranoid type too, so value of getting hallucinated outputs and having it critique my writing through the tapped GPT web interface isn't that high.

If you could read all of these outputs and compare against each others(thanks glandium-san and sorry for bothering), you'll just see all of it has pros and cons and that they compare. They're not worlds apart, in fact I think GT did best in this instance probably from massive accumulation of edge case fixes or some other non-scaling improvements behind it.

Like I said, it's an interesting phenomenon - it could be UX advantages, or friendlier tones, latency, personally I think it's dataset imbalance or could be numerous other things. But when it comes to actual literal translation performance, um, it's just an MT.