Comment by Roark66 | Hacker News Reader

Comment by Roark66 | Hacker News Reader

Great stuff, I skimmed the article searching for some table showing a breakdown of content by language, but I haven't found one.

I hope there is a lot of text in languages other than English. As for example in my language (Polish) current SOTA models are very deffiecient. I have wondered why is that considering companies like (not at all)OpenAI claim to train on large datasets including in my language of interest. It turns out (and I learned this just yesterday) they used LLM translated English content that that used as other language training data. They used Azure translator which itself is a transformer model to generate content for gpt-3.5 for example. Also, I bet there is a lot of poorly machine translated content in their supposedly "original" data.

The result? You can use chatgpt to write you an email of any kind in English and you can copy/paste/send immediately. Try doing that in Polish... It will make sense, but the language used will use bad tone (too familiar in a business setting), bad words(words that exist, but no real person would use) and sentence layout that just plainly feels weird. I suspect this is even worse in many other languages.

While having multiple languages makes a model more versatile and appeal to a wider audience, it actually significantly increases the memory required to run the model and thus limits other aspects of the model.

Optimally, a Polish audience should try to create a Polish trained model.

As it stands now, most advanced models, like gpt are multilingual, but are noticeably less capable in non-English languages.

Having every model re-trained in each language is a certain path towards having any non-English (or at most a couple of other languages from countries with big pockets, like Chinese) language model be always massively behind - the resources required to train a model are huge, you can't expect e.g. the Polish community (plus anyone else) to replicate every good English model that comes out. GPT4 is less capable in Polish than in English, but probably much more than any Polish-specific model ever trained - and I suspect the gap is bigger than that with the best non-GPT4 English model.

Furthermore, I think you are exaggerating the memory issue of multilingual models significantly. Especially for languages using the same (Latin) script, the additional characters to care about are very few. Also a significant part of the vocabulary and language fall into a few buckets, so training a joint model makes all the sense in the world - much like an Italian native speaker could likely study a scientific text in Spanish and understand its content, even without speaking the language.

The memory impact comes mostly from having bigger embedding layers that have to account for vocabulary in many languages (the most problematic case being Chinese and Japanese, with their huge set of tokens). But even there, the largest vocabularies in use are maybe of size 100k (vs. about 30k for English-only), with a hidden dimension of 4k that makes for a total of 400M parameters. It's a lot, but a drop in the ocean of 100B+ parameters (or 1T+ for GPT4) we're seeing today.

P.S. Answering to GP, I think the Pile is English only, though - or at least, models on HuggingFace trained on the Pile, like the various Pythia models, are tagged as English only.

The biggest difficulty people have IMO when trying to train language models in non-English languages is that there is not enough text written in these other languages to select a big good quality dataset.

Also, there are lots of (poorly) machine translated websites in Polish... So any dataset that contains web crawl will have precisely what I'd prefer not to have.

Ideally I'd see either the national gov, or the EU to invest money into creation of more high quality datasets in all EU languages.

So when I point out the failures of for example chatgpt in my language I do so while being amazed it can generate and understand Polish at all.

Also regarding multilingual model size being larger. I never heard this before, but it seems logical. I have heard models gain extra performance on English tasks when they are trained on other languages too so there is a benefit to adding multilingual datasets.

The Pile and Red Pajama are primarily English language datasets. If you want something multilingual, I'd suggest having a look at the Bloom dataset https://arxiv.org/abs/2210.14712

Reply on news.ycombinator.com