Hacker News Reader

Comment by kvam | Hacker News Reader

As a Norwegian this sounds like a mistake. Who will use this LLM? Where? For what? The underlying data could be made more easily searchable and digestible for agents in general if the goal is better knowledge of Norwegian culture.

View on HN

I agree in principle.

That said, they are quite limited in what they are allowed to share of in-copyright works, and nb.no is a fantastic resource as it is (though you'll need a Norwegian IP address for too much of it - it's one of th main reasons I maintain a VPN) - if they are allowed to make it accessible there, it'd be great.

But they also have vast amounts of out-of-copyright data that I hope they'd make more easily accessible...

Hard disagree. This is the first step not the last and proves to other countries that this can be done.

This model is going to start miles behind the frontier and the gap will only grow.

Why would the gap grow? There is no more training data to acquire, frontier model are training on the entire internet. Everything from now on is just fine-tuning.

Your statement assumes training data is the only thing that matters for the big players, while not considering it limiting for the small Norwegian model. That’s a fallacy.

Nowhere in the article does it say the Norwegian LLM will train _only_ on Norwegian data.

Orders of magnitude less compute for pre- and post-training than the the frontier labs.

Exactly, if there's one thing transformers are good at it's translation. One I've found particularly nice: any question ChatGPT can answer in English it can answer in French. I'm assuming Norwegian too. So there's no point.

There's quite a bit more to culture and language than just being able to have transformers come up with believable language and/or dialect.

They're only good at it because they were trained on massive amounts of English and French data.

Not really true.

Both Claude and ChatGPT can translate into minor dialects of Norwegian they will have seen very few works in because very few printed works exist in them.

E.g. I've tested both my local spoken dialect, which is rarely written, and a sociolect used by a 1970's Maoist group consiting of a few hundred people, where most of the printed material consists of novels from a couple of ex-members that became authors.

In the latter case, it claimed to not know, but was able to get a good match from just a description.

I also just had it ape Norwegian orthography from the 1910's by having it look up the rules and translate a text it had first translated from English to modern Norwegian, and it did just fine.

They will have seem some work in these dialects, but mostly it transfer really well to know related languages (English, Dutch, German, Swedish, Danish, roughly form a continuum from least in common to most in common with modern Norwegian; they all share vocabulary and significant parts of grammar with Norwegian), and then a relatively limited exposure to Norwegian itself is sufficient to do fairly well.

They're also really good at "style transfer" of text in the form of tweaking orthography, word order, and minor grammar changes from descriptions and examples.

(incidentally, the latter is one way of getting an LLM to sound a lot less like an LLM)

This is all true, but I assumed the original posters were talking about cultural knowledge, not linguistic correspondences.

To do translation well you still need cultural knowledge. (E.g. the particular modes of specific kinds of legalese, or slang and the nuances of social class, etc)

I think it's not that this knowledge isn't present in the model somewhere, but probably more that it gets killed by instruction tuning for US corporate values.

The point is that norway willl have its own LLM. And will not have dependencies to another state or private company. The goal is not to be the best model. But to have a model that include more Norwegian data then other LLM and that it's not screwed against other sources.

But what does that give you? If the model is far less capable? What will it do for you with that Norwegian data, that a better model could not do with better search or context?

A model has many dimensions. You can't have them on one scale from good to bad. The model will most likely be poor at coding. But will give better answers about Norwegian cultur. I assume the tone of voice will be (by default) much closer to how Norwegians talk and write then what we current see from model from the US. They seem to be a bit to much.. Norwegian people are a bit more down to earth

Other better models cant use the training data this model have...

Yes transformers are great at translation as that is their purpose.

LLMs are not great at preserving cultural uniqueness and diversity. Take how “delve” has reentered the lexicon because the human assessors for pre training dialect of English uses “delve” a lot.

There is a lot of benefits to training specifically for a unique culture with unique norms to preserve the culture as we increasingly rely on LLMs.

https://www.scientificamerican.com/article/chatgpt-is-changi...

Model can speak Lithuanian too, but with a Russian accent which is a big taboo for us.

I wonder, can you ask it not to do that first, and check if that still happens?

Definitively not the case in distilled models.

There is a lot more to it than literal translations. Even if an american can talk Norwegian it doesn't mean that they get the cultural context right.

"Oh yeah after you drive 2 hours home from work your wife and kid will greet you with some delicious pie" doesnt work so well even if its in Norwegian.

Reply on news.ycombinator.com