AI translates Hokkien, an unwritten language, for the first time
tech.fb.com
tech.fb.com
As a person who grew up having Hokkien as my first language, I've feared that Hokkien might go extinct over time. Having a formal English and Mandarin Chinese education had pretty much started to erase my Hokkien knowledge. I'm glad that AI technologies can be used to preserve this language.
The way to preserve a language is putting humans in the loop. Creating content in that language; interestingly more and more shows are produced in Taiwan in or using non-Mandarin languages as a political way to mark a difference with the big neighbour. And having government support, notably at school (at young age) by allowing partial or total teaching in the language to be preserved.
To put it another way, in 10 years when I visit China I could probably have an in-depth conversation without ever knowing Chinese.
I’d the Chinese can come to a university and never need to learn English, they won’t. Your parents and community will teach out to speak and that’ll be the end of it.
It’ll be closer to the Tower of Babel scenario. Every community will have their own language and dialect and the AI will just adapt. If a solar flare takes out electronics we won’t be able to understand each other.
Alternatively, the AI will teach us to speak or an implant will teach us similar to the matrix.
Those are the scenarios in the next 25-30 years I see.
Doubt it. When you learn a language you learn a lot more than the translation, you learn thinking and connotations that require a broader understanding that never get captured regardless of the translation quality, because they require knowing history and seeing previous uses in context that can't be captured in the translation.
"Good" translation will do for language what wikipedia has done for knowledge. Everyone will get a superficial understanding, some will pretend that reading off a definitions is the same as actually knowing something, and the world will get a lot more shallow, empty conversations.
In some languages, DeepL has near perfect translation to the point you cannot really tell if it was translated.
preposterous. having an indepth discussion with a foreigner requires not only the language but also the understanding of the cultural context.
It's also counterproductive to let humans learn a language of limited content resources and use cases.
Taiwan people are highly educated and urbanized. It's much harder to use Taiwanese in Taiwan compared to High German in a Pennsylvanian Amish village.
I don't know how to express clearly in Taiwanese "GPS in my neighborhood has a 100x lower accuracy because of radio interference" or "move this MOSFET up by 15mm to balance the PCB thermal stress". If you still have to switch to Chinese or English from time to time, why not just use the popular languages?
Even Japanese, a language used by 125 million, has similar issues, my Japanese coworkers frequently switch to English during technical discussions.
Yes, it is possible to do everything 100% in Japanese, with the only English being the keywords of whatever programming system you are working with. However, that is more the exception than the rule, especially in larger teams that need to work overseas.
[0] Creator of Super Smash Bros. and Kirby
[1] "Masahiro Sakurai on Creating Games"
There are of course aesthetic/logistics reasons behind "code the stuff in English" (if only cuz your code is going into an ASCII codepage, and ... yeah, sharing). But the language used in teams is pretty company-culture dependent, and "we do all of our work in English" lands in a very restricted set of companies. Probably Finance is the one where that culture is there, but most tech companies.... there are higher-than-average english language levels in these places, but if there's no other English-preferer (foreigners, but also returnees or people who just like English a lot) in the room? Not happening
Your example for difficult use is particularly apt in that you’re choosing to focus on specialized technical examples, which often default to international lingua franca anyway, often English.
I doubt that day to day use of Taigi in Taiwanese communities is as rare and difficult as you say. Maybe in highly educated and urbanized Taipei, but have you even been to the southern countryside?
El GPS en mi barrio tiene precisión cien veces peor debido a la interferencia de radio. mueve este MOSFET 15mm hacia arriba para balancia de estrés térmico a la placa.
I believe you can "invent" a Taiwanese sentence to mean the above, but there is no consensus among Taiwanese speakers in how to say them, so they would need your explanations of what your chosen words mean. If you borrow Chinese words, your sentence will be no much different from the Chinese sentence.
For your question -- yes you can scrape by with only Taiwanese, just like you can live in some areas in the US speaking only Spanish. But to do anything more, like riding a train to another Spanish speaking area, you could meet a conductor who have to open the translator app for you.
I also lived in the Guishan/Linkou area for a year and heard a lot more Taiwanese than Mandarin in day to day life.
This is really not common and if anything it's something unheard of to me. I work in an English speaking company in Japan and most of my coworkers (who are fluent enough to speak English in technical conversations) would instantly switch back to Japanese to talk about technical things between them if there's no foreigner involved in the conversation. I've seen the same thing happen in my wife's company and other companies too. This is on top of the fact that the level of English education in Japan is very low (unfortunately) and these people who work in English-speaking companies are very much the exception. I don't think I've ever seen a single Japanese person favor using English over Japanese for technical discussions if they were ever given the choice.
I struggled with Google translate because it’s like 40% accurate at translating technical related stuff.
Are you Japanese yourself? If not I don't think it's strange that they would adapt their way of speaking with a foreigner, especially since most technical words in IT are coming from English anyway. For other fields, health for instance, it's totally possible to never heard English in months/years. Japanese is well alive, and English is more a social marker than anything else. Most Japanese have very bad command of English if at all and can live their whole life never using it.
You must live in a kind of weird cocoon because in Japan nobody switches to English for technical discussions because they cant even speak English in the first place.
It's a chicken and egg problem. If you don't do technical discussions on Japanese now then you won't do them in the future either. You have to consciously start doing that and then eventually you'll do it that way as a preference.
>why not just use the popular languages?
Because you will then take over their cultural baggage. Look at English and the internet. Americans are outnumbered and yet it's expected that people follow the norms of American culture online.
If you speak German on the same websites then those norms more or less disappear.
For Japanese, it's opposite. People aren't good at English so Japanese tech writing/talk is everyone's preference. Also Katakana helps a lot to import foreign words. Top people learn English, import words, wrote their text, and talked in Japanese.
Now Japanese texts are getting not popular (due to not profitable) and many English texts are very easily accessible thanks to internet and it's lingua franca. So finally many Japanese are going to learn English to catch up.
This isn't true at all. Taiwan is highly educated and urbanized and nearly everyone can understand Taiwanese. Millions of people speak it natively and everyone else has encountered it in media, day-to-day life and also in school in the past 15 or so years.
Taiwanese/Hokkien have limited content resources because of conscious decisions of previous governing bodies in China, Singapore, Taiwan (not sure about Malaysia). If they had been allowed to flourish, actively promoted and native speakers were taught how to write, it would be a lot different today. The premise of the article is false. Taiwanes/Hokkien has had a standardized written form since the 19th century. The problem is most native speakers do not know it, so really what it suffers from are low literacy rate. I communicate in written Taiwanese via text quite often.
Over the past ten years or so, the government in Taiwan has been trying to promote native language literacy, but with so-so results due to what I see as poor pedagogy.
> I don't know how to express clearly in Taiwanese
Many heritage or home speakers probably feel the same way. There are strong social stigmas against using Taiwanese in academia or people seeing it as a crude language. But if you are a native speaker of both Mandarin and Taiwanese, it isn't too hard to learn the written form if you want to.
I'm not a native speaker of Taiwanese, but I can make somewhat intelligible translations of them. A native speaker who has learned to write Taiwanese and knows English could do a better job.
> GPS in my neighborhood has a 100x lower accuracy because of radio interference
Tī goán chhù-piⁿ in-ūi tiān-chû kau-jiáu só͘-í GPS ê cheng-chún-tō͘ khah pháiⁿ chi̍t-pah pē.
> move this MOSFET up by 15mm to balance the PCB thermal stress
Chit MOSFET ūi-tio̍h chè-ap PCB ê on-tō͘ ài khǹg kòe-khì siōng-chē 15mm.
> Even Japanese, a language used by 125 million, has similar issues, my Japanese coworkers frequently switch to English during technical discussions.
That's interesting. I have only experienced preference for English technical vocabulary, but never switching of languages. Even for native English speakers they need to have familiarity with the subject or those technical words are unintelligible to them.
[1] (ASL speakers claim to 'think' in sign)
In this sense, we would exchange information that was of a higher order than language. For example, a complex idea like "the location where I will meet you for lunch this afternoon" would be independent of the verbal language we use it to express it.
We can see a similar (albeit more limited) exchange of language-independent structural ideas in mathematics. We distinguish between "numbers" (the idea of a quantity) and "numerals" (the symbols we use to describe that quantity). For example, `III`, `three`, `3`, `5 - 2`, and `the number of complete revolutions made by rotating 6*pi radians` are all ways of defining and representing the same common idea — "the number three". Going back to the previous example, imagine if an idea "where I'll meet you for lunch today" could be universally and unambiguously shared in a similar way, and you'll get a rough approximation of what perfect telepathy and perfect comprehension would be like.
Every time that I have a word on the tip of my tongue I have an idea in mind that I wish to express, but I simply cannot remember the word to convey the concept. That is to say that I've already thought of the idea and am now merely looking for a way to express it.
However, I think that "thinking out loud" in your head has benefits. It makes the idea you wish to express more concrete and lets you iterate on it more easily. Think of it like writing down an equation when doing math: it gets easier to reason about it when it's expressed externally. I think ideas and thinking verbally in your head work the same way.
So, The idea of thinking without directly using language is less foreign to me than many. Nevertheless, I'll still sometimes use 'words' internally to reason things through which are too complex to solve intuitively, or when enacting a procedure I learned from someone else.
It's not clear to me that the engrams you imagine wouldn't just be a symbolic language under another name. I suppose if they were continuous in some vast highly dimensional space ... but even then one could discretize them at the resolution of distinguishable ideas and it's a symbolic language again. :)
Going extinct would be sad, sure, but I'm not sure that the effort required to not make it so would be worth it. Putting children through thousands of hours of education in another language, for what exactly? If I have kids, I'm not sure I'd be willing to put them through all that work for a nearly dead language. They're not gonna use it to speak with their peers, or to open up communication with another group of people, it's just preservation for the sake of preservation.
After all, the nineteenth century saw the consolidation of national languages and elimination of regional dialects in the name of fostering nationalism, which continued in the twentieth but also saw the revival of languages like Irish or Hebrew for new nationalisms.
Also, the differentiation potential is somewhat limited due to the fact that the majority of Hokkien speakers lives in Fujian province on the mainland (Hokkien = Fujian) and there's some preservation work going on there as well. E.g. Xiamen University's Speech Lab had a working demo of spoken-Hokkien-to-written-Mandarin translation in 2018, although the link shortener they used has since suffered from link rot. https://speech.xmu.edu.cn/2018/1215/c18169a359542/page.htm
Also, Hokkien interest is present even outside of that geopolitical flashpoint:
https://www.todayonline.com/singapore/stories-behind-tiktok-...
Of course Mandarin, is more "useful" in a purely pragmatic way but languages are social things, they're tied to culture, they help create relationships with other people who speak those languages (I have plenty of Teo Chew friends who have made very good friends with others from different country because they are kakinang) and speaking multiple languages is always worth keeping.
Plus even besides this, having a multilingual home, regardless of the usefulness of the languages that are spoken, is associated with a lot of cognitive benefits for children so the thousands of hours learning a different language are useful.
The day every person on Earth speaks some shared common language (ideally one so straightforward that it can be learned by children within a year or so) will be a day I'd celebrate as a monumental milestone in the development of our species.
It's fine if people know other languages too, but having that shared global one is vital.
I'm happy to lose innumerable untranslatable phrases and cultural understandings in service of this.
Cultural spheres have always managed to come up with a lingua franca that enabled them to exchange ideas. English fills that role currently and will probably endure to dominate. Even if something goes monumentally wrong with the Anglosphere, it will endure until another language manages to step up to that role.
Children are perfectly able to learn multiple languages within a few years by pure immersion. I can't see what further optimization here would achieve.
Still, judging by the events of the past, languages that are not sanctioned by some state will probably all die out by the end of this century. Further erosion is very unlikely though. The language of any country with at least, say, 50 millions speakers is probably safe.
There are Hokkien-speaking people in Hong Kong and China's Guangdong province. They write Hokkien in Chinese characters mixed with pinyin for these without a corresponding character.
"if "pure, unadulterated spoken vernacular Taiwanese" were written exclusively in Chinese characters, with minimal use of Mandarin phrases, over 25% of morphemes would have no character, about 25% would have arbitrarily selected (yet more or less conventionally accepted) characters that are homophones or near-homophones, 10% would be written using characters exclusive to Hokkien, and 40% would be written with characters that have the correct sound and meaning. However, in more colloquial styles of Taiwanese Hokkien, the proportion of morphemes written with conventionally accepted characters would drop even lower than 40%."
Calling languages like Hokkien “a slightly different dialect” is really underselling it. These are totally different languages, as divergent as German is from Portuguese.
I understand the concept of formal literary languages but in this case, having literary Chinese as your "written language" feels like an anachronism. Like treating Latin as literary language in medieval Europe: ultimately disenfranchising to common people and something that needed to be set aside.
That comparison is a stretch, as Latin and English are both written alphabetically, while classic and the various modern chinese dialects/languages are written logographically.
It's a delightfully complex topic! The headline is a bit misleading, as wikipedia has an entire article on written Hokkien - as you note, it isn't standardized. https://en.wikipedia.org/wiki/Written_Hokkien
Hokkien, specifically the version mentioned in the article (Taiwanese Southern Min) has two main writing systems: Chinese characters and Latin letters based romanization. Heck, that's actually more than most languages in the world! The Tâi-lô[1] system, a slight modification of Pe̍h-ōe-jī was recently standardized by the government. On the other hand, Chinese characters can be used to write Taiwanese, not just Classical Chinese or Standard Chinese (Mandarin), save for a few words that are often written using Latin or bopomofo letters. It's true that a large number of speakers have not been taught either of those writing systems, but it does not mean they don't exist. I've already met someone which taught me few words in Hokkien (the infamous chia̍h-pá--bōe 食飽未) and write it in Chinese characters for me.
I'm working with various Sinitic languages, and Taiwanese is far from being the worst when it comes to available resources in written form (for my use cases dictionnaries).
[1] https://en.wikipedia.org/wiki/T%C3%A2i-u%C3%A2n_L%C3%B4-m%C3...
In case of Taiwanese, there is a small but existing corpus of literature, as a 5 seconds Google/Wikipedia search shows[2][3].
[1] https://aclanthology.org/O16-1026.pdf
[2] https://zh.wikipedia.org/wiki/%E8%87%BA%E8%AA%9E%E7%99%BD%E8...
[3] https://zh.wikipedia.org/wiki/%E5%8F%B0%E6%96%87%E5%B0%8F%E8...
Cantonese survives and is thriving because it dominates as a spoken language in education and in public. Even so, only few people are familiar with any of the romanization systems for Cantonese. In Taiwan, Hokkien and other languages are not so established, and were even suppressed for some time. Therefore, it's not so surprising that even fewer people know about writing systems for these.
[0] Characters derived from Chinese. Each character has both an embedded meaning and one or more readings. They look like this: 漢字
[1] Characters that spell out whole syllables. The glyphs are typographically derived from kanji, but very simplified and without meanings. Due to medieval Japanese gender politics, there are two of them. They look like this: ひらがな and カタカナ.
[2] The characters you are using right now.
This is only because the plan for simplified characters died midway through. There was a second round of simplifications that would have gone even further with talks of full phoneticization if that succeeded. However, the second round of simplification was simply too unpopular.
> While Lu Xun is now taught in all mainland Chinese schools, they conveniently leave out the part where he advocated for leaving characters in the past.
They definitely don't. His arguments and polemics about this are pretty well known in China. There are a variety of secondary and undergraduate courses that cover his collected works, with that specific quote showing up all the time. This is also a hot topic that repeatedly shows up in all sorts of Chinese online forums.
Lu Xun, like the other scholars of his time, was extreme in the other direction. He believed that it was fundamentally impossible for China to advance without a phonetic script and that China was doomed and it was impossible for the Chinese citizenry to achieve literacy. His arguments are generally discarded today because he said things that have been soundly disproved.
It is worth noting that many early 20th century scholars had a fanatical devotion to anything Western. They contorted Chinese to make it as close to Western languages, especially English, as much as possible, adding new pronouns, following Western grammar patterns, etc. Indeed, some early 20th century Chinese articles are quite awkward to read without an English background.
Leaving aside whether characters are "good" or "bad," there is an oft-raised, but mistaken, view that using a phonetic script for Chinese would not change the language. After all it is but a different medium for the same information.
But that's not true. There remains a level of diglossia in modern Mandarin that makes some modern works difficult to decipher when read out loud (yes this is true even if we leave out Classical Chinese). Using a phonetic script to write Chinese would necessarily involve a change to the language, not just its transmission. Whether that's worth it is a separate question, but it is a change.
OK, {{citation-needed}}. FWIW, my source is two decades talking to locals. I am yet to meet anyone aware of these aspects of his position save academics, therefore I am genuinely interested in why you claim this is "definitely" false.
I agree with the rest of your points.
But as for sources...
It shows up on high school tests. E.g. a 2022 high school test from Dalian https://www.51jiaoxi.com/doc-13250120.html (there's a paywall but it's client-side you can remove it with your browser's DOM inspection tools)
> “胡适提出的文学革命其实是一剂泻药。而陈独秀、钱玄同、鲁迅等人还嫌它不够猛烈，又以更激烈的态度提出新的口号，于是‘废除汉字”‘将线装书统统丢进茅厕里”“不读中国书”等种种骇人听闻的说法便一个接一个地出来了，而且到现在还有人为之猛拍巴掌。”该评论指新文化运动（ ）
> "The literature revolution championed by Hu Shih in reality was a laxative. Yet Chen Duxiu, Qian Xuantong, Lu Xun and others thought it was still not extreme enough. As a result, "Get rid of chinese characters," "Throw thread-bound books [the usual book binding for classical texts] into the toilet," "Don't read Chinese books," and other shocking sayings came out one after the other, and even now there are people who applaud such sayings." This comment points out that the New Culture Movement ([a series of multiple choice answers follows])
There are literally dozens if not hundreds of threads about this on Zhihu, Baidu Zhidao, etc. (https://www.zhihu.com/question/36621372, https://www.zhihu.com/question/525670306, https://zhidao.baidu.com/question/2201546777143697068.html, etc.)
This is a pretty far cry from "conveniently leaving out."
I was under the impression that most of the debates about moving from Chinese characters to alphabetic writing happened in the pre-PRC period. For example, Lu Xun supported Latinxua Sin Wenz[1] in the 30s. These proposals failed for a variety of reasons. Simplified characters were introduced in the 50s. Pinyin was also introduced in the 50s, but unlike previous latinisations meant to replace the Chinese characters, it was only ever intended as a teaching tool. I think there was still a thought to replace Chinese characters with alphabetic writing at a later stage, but, in practice, it pretty much died in the 40s.
> 漢字必須改革，漢字改革要走世界文字共同的拼音方向
> Chinese characters must be reformed, and that reform of Chinese characters must follow the global trend of phonetic spelling (pinyin)
This was swiftly followed up by 1958 by a pronouncement from the head of the government committee on character reform (http://www.gov.cn/test/2008-03/07/content_912608.htm)
> 我们认为：汉字总是要变的，拿汉字过去的变化就可以证明，将来总是要走世界共同的拼音方向。
> We believe: Chinese characters inevitably must change. We can use the changes in Chinese characters of the past to prove that in the future, this must follow the global trend of phonetic spelling [this is almost a carbon copy of Mao's words]
This sentiment continued through the second round of simplification. The People's Daily (the usual mouthpiece of the PRC) wrote an article in 1977 (https://www.laoziliao.net/rmrb/1977-12-20-1) in concert with the beginning of the second round of simplification explicitly describing character simplification as setting the stage for phonetic spelling.
> 毛主席指出，汉字的拼音化需要做许多准备工作；在实现拼音化以前，必须简化汉字
> Chairman Mao has stated: the pinyin-ification of Chinese characters requires a great deal of preparation. Before we can achieve pinyin-ficiation, we must first simply characters.
But by that time literacy rates were sufficiently high that there was considerable backlash against the second round of simplification and it was withdrawn, first informally and later formally. Had it succeeded, no doubt a fully phonetic script would've been at least on the table for discussion.
> 至于汉字的前途问题：它是永远不变还是要变呢？它是在汉字固有的形体范围内变化，还是被拼音文字所代替呢？它是为拉丁字母式的拼音文字所代替，还是为别的形式的拼音文字所代替呢？这些问题我们现在还可以不忙去作结论。我们认为：汉字总是要变的，拿汉字过去的变化就可以证明，将来总是要走世界共同的拼音方向。而且可以说，世界各个民族的语言和文字，将来总有一天会逐渐接近和统一。但是这些都不属于当前文字改革的任务的范围，因此我们今天还不必加以讨论。
> As for the question of the future of Chinese characters: will they never change or will they change? Will the change be restricted to the scope of their current form, or will they be replaced by a phonetic script? Will they be replaced by a phonetic script based on Latin characters, or will they be replaced by another form of phonetic script? For now, we will not rush to any conclusions on this topic. We believe: Chinese characters inevitably must change. We can use the changes in Chinese characters of the past to prove that in the future, this must follow the global trend of phonetic spelling. Moreover it can be said, the languages and scripts of all the people in the world will one day come together and unify as one. But these are not within the scope of our current reform tasks and as such today we have no need to discuss them further.
The committee head's pronouncement was in fact essentially paraphrasing and nearly copying large parts of Zhou Enlai's speech. So I would not characterize his speech as closing the door on a phonetic script, only setting it aside for now and leaving the door open for a future date. From Zhou Enlai's speech:
> 至于汉字的前途，它是不是千秋万岁永远不变呢？还是要变呢？它是向着汉字自己的形体变化呢？还是被拼音文字代替呢？它是为拉丁字母式的拼音文字所代替，还是为另一种形式的拼音文字所代替呢？这个问题我们现在还不忙作出结论。但是文字总是要变化的，拿汉字过去的变化就可以证明。将来总要变化的。而且可以说，世界各个民族的文字形式将来总有一天会逐渐统一，甚至语言最后也会逐渐统一。人类的语言文字发展的最后趋势是逐渐接近，到最后也许就没有多大区别了。
> As for the future of Chinese characters, will they not change across the eons time? Or will they change? Will they trend towards changes based on the shapes of characters themselves? Or will they be replaced by phonetic script? Will they be replaced by a Latin character-based phonetic script, or will they be replaced by some other form of phonetic script? As of now, we are not yet in a rush to make conclusions. But writing must change. The past changes of Chinese character can prove this. In the future, there must be change. Moreover it can be said, the scripts of all the peoples of the world will one day unify as one, indeed language itself will gradually unify. Humanity's language and script development's ultimate trend is to gradually become closer. In the end there may not be much of any difference [among different languages].
And as we can see that rhetoric about a fully fledged phonetic script would indeed come back once the initial reform plans were finished.
Only with the failure of the second wave of simplification does it seem that future plans for a phonetic script finally disappeared from the conversation.
not in latin countries which had languages and dialects not too far from Latin in the first place
However, a significant part of Hokkien is from classic Chinese which has written symbols, and the other part is from some local dialect in Fujian.
During that same presentation, we also know that Zuckerberg lied about the metaverse and how characters are able to move their legs (it was filmed).
We can try it out right now. Given all of this, I am much more reason to believe this more than Zuck's metaverse claims.
People say that Hokkien is a dying / dead language, but the way I see it is that it won't die out completely in my lifetime so it's still worth learning.
[1] https://en.wikipedia.org/wiki/International_Phonetic_Alphabe...
AI -> General AI
ML -> AI
Anthing-not-deep-neural-networks ML -> ML
The spread of concepts across languages has provided meaningful benefit over thousands of years.
People who are multi-lingual also speak of how after immersing in one language or another they start to dream and think in a new language and this adjusts their thought patterns.
The affect of our dominate language on us is not neutral and different languages bring different benefits and framing.
I will buy it.