Hacker News Reader

Comment by clankyclanker | Hacker News Reader

This is precisely the case where ChatGPT (or any LLM) will have trouble that a layperson would have trouble validating: Toki Pona has a limited document corpus and requires contextual interpretation obvious to experts.

In short, expect hallucinations and other lies.

View on HN

In this case, I validated against the hand-translated example in the main link. It did fine.

This is probably easy mode for it, given the tiny vocabulary at hand, even though it has surely seen very little Toki Pona in the wild. Large language models are extremely good at translation. I’ve personally tested ChatGPT-4 against Google Translate and other translators on some hard translations (languages that are not similar to English, such as Chinese), and it has consistently provided better translations than Google Translate in my opinion. As an example, for validation purposes, news stories that have been reported worldwide make things easier, since you can establish the facts of the translation by hand, and then compare what comes out.

What does your comment actually add to the discussion? Yes, ChatGPT can be wrong. So can Google Translate!

If you’d like to provide a clear example of ChatGPT-4 (not ChatGPT-3.5) making up a Toki Pona translation that is completely disconnected from reality, please do. That would be legitimately interesting conversational material. General cynicism isn’t.

ChatGPT absolutely has limitations, and translation is a hard problem, even for the industry-standard tools. Crossvalidation with multiple translation tools and ideally humans is important for any translations that actually matter.

Since LLMs' only valid uses involve entertainment and creativity, and Toki Pona is more or less a recreational language without serious application, I think they go together like two peas in a pod.

>only valid use...

Well. That's an opinion I suppose.

-edit- After posting, I decided this was too snarky/low value on it's own. So to add:

I personally get quite a bit of value out of it as an ad-hoc German tutor and as one-among-many tools for research and learning on new topics for which I am not already an expert.

I've also used it to help me make and troubleshoot very simple code in languages that I am not an expert in.

I personally have trivially found lots of uses that probably all cluster around "educational" to one degree or another.

Is it perfect? Absolutely not. Does it hallucinate/get things wrong? Absolutely. Is it useful despite those limitations? Emphatically yes.

This idea that its "only valid use" is entertainment doesn't even make sense to me. "Entertainment" is perhaps where I find it the most deficient relative to other current options.

  > In this case, I validated against the hand-translated example in the main link. It did fine.

Presumably that hand-translated example was also part of the training data.

Not likely. The article says it was posted 1 year ago. ChatGPT’s training data cutoff is September 2021, so no, that would not make sense.

I've done a fair amount of translating Toki Pona based on my own sentences, and based on recent online posts. None of that would have been in the training set. It does a good job. Since Toki Pona translation is "lossy", you can even prompt it to give you multiple interpretations and discussion.

Another challenge, is that toki pona requires a lot of context, not only of previous sentences, but visual and communal. For example, I can say 'soweli lili' and point to a cat, then in all further conversations 'soweli lili' will mean exactly a 'cat' until specified.

I could use sentence that could mean literally hundred thousand of different things, but if I explain it properly once, you are expected to keep it as context.

ChatGPT4 will struggle to keep all context as it will surely accumulate.

Lies are typically done on purpose and in this example used to indicate intent, when there is none.

Nope:

https://news.ycombinator.com/item?id=35483823

Reply on news.ycombinator.com