Consortium launched to build the largest open LLM | Hacker News Reader

Consortium launched to build the largest open LLM

utu.fi

117 pointsoutrun8653 comments

I recall a post on HN a while back about the EU planning to invest billions in AI. I can't find the HN link but found this source: "annual investment volume of €20 billion over the course of the digital decade"[0]. I wonder what happened with that money. It seems like using it to train large scale models would be a relatively effective use of that money.

[0] https://digital-strategy.ec.europa.eu/en/policies/european-a...

It's looking for projects like this, so you end up with silly things like the University of Turku announcing it's powerhouse of LLM expertise (?) will train the "largest open model" in a consortium with (??).

The only concrete factual detail is they'll train on euro-focused data

15 million GPU hours is interesting, llama 2 had about 1.7 million hours for the 70B (3.3M for all). Their GPU hours are on LUMI which is AMD MI250 GPU, which Mosaic reports is about 75% of the A100 (https://www.mosaicml.com/blog/amd-mi250) but AFAIK hasn't been tested at this kind of scale. so, validating if the MI250 in a cluster can effectively train a large scale LLM will be useful. not that it really matters to any other group, you won't get these in the cloud anytime soon.

either way - say that they have about 10M A100 hours. if they use half their budget for the biggest model as FB did with Llama (rest on testing / smaller models as they scale), then they'll budget 5M A100 hours for the biggest model, or 3x llama-2's 1.7M A100 hours (2k A100 for 21 days).

However, why wouldn't Llama-3 also be 3x bigger than Llama-2? FB's H100's are coming online; they apparently have 4k H100 in a cluster (https://www.nextplatform.com/2023/09/26/meta-platforms-is-de...), and each H100 is about 3x the speed of an A100 (https://www.mosaicml.com/blog/coreweave-nvidia-h100-part-1). So if they provided the same 'calendar time' of 21 days over a single cluster of 4096 H100, that would be about 6.2M A100 hours for a 70B model.

there is an entirely separate question of, LLM's are functions of compute _and_ data, will they also find enough high quality _and_ kosher tokens in their low resource languages?

If this gets enough of an official stamp of approval from the EU, then access to enough tokens in smaller languages may not be a big issue. For that matter, even for OpenAI it's likely largely down to how much they care to negotiate access.

E.g. for Norwegian the Norwegian national library and state archives sits on at least a couple of magnitudes more of digitized material in Norwegian than OpenAI appears to have found Norwegian material for GPT3.

How much exactly would depend on willingness to give access to still in-Copyright data, but it'd still be far more even if you only give access to what is currently on their website.

And ChatGPT is well versed enough in Norwegian even without that to be able to convincingly use several regional dialects that are rarely used in writing (I've tested several) so it won't take all that much.

But frankly being able to ask it about the contents of these archives would be amazing because they include a couple of centuries of newspapers and almost every book ever published in Norwegian.

The level of digitization will differ, and Norway was early with that, but most countries have depositary requirements for printed materials, and countries with smaller languages like are often more anal about them...

Does training an LLM on a variety of different languages improve its reasoning and understanding in other languages?

For example, if GPT-5 (or whatever next gen model) was trained on a staggering amount of high quality Norwegian tokens that talk about medicine, city life, romance, maths, etc... would it be able to apply those concepts & learnings to other languages?

I'm trilingual and have ~100% of transfer of knowledge/reading I learn/read in one language into the others, curious to see if LLMs are like that.

I think how well it handles Norwegian despite the small number of Norwegian tokens present for GPT3 (I'm not aware of up to date overviews of token distributions in the training data) is a good indicator that a lot of it does transfer.

the 'low resource languages jailbreak gpt-4' paper (https://arxiv.org/abs/2310.02446v1) shows that the models, when prompted in a low resource language, return highly coherent and sophisticated responses. it's possible that RLHF does not do a great job here but the base models are clearly cross-applying intelligence from high resource languages to low resource languages.

I think data quality, not just largeness, should be a goal. A huge and excellent dataset is something only a massive organization can afford.

Someone needs to gamify cleaning up data.

What is the bottleneck in building the "largest LLM" today for any interested party? AI expertise? Training data? GPUs?

Money?

Plenty of people have money, but they don't all have world class LLMs. Clearly money isn't the only problem.

Not everyone who has money wants to build one .

It is just a shiny new toy, they may deem there are better uses for their money

GPUs and electricity. GPT-4 reportedly cost 100M$ to train.

I'd say GPU and people trained in knowing what to do while training. There is a lot of money in the world, as we see Microsoft and other companies handing it out. Power even, if you build in the right place can be cheap, see bitcoin mining.

Very much agree... I think people underestimate how finicky these models are to train. So much so that one of the big announcement around GPT-4 was the fact that OpenAI found a way to make it smoother and more predictable.

As evidence by this 114 page log of how engineers resolved problems that came up during the training of OPT-175, which lasted for several months. https://github.com/facebookresearch/metaseq/blob/main/projec...

It has to be more than just that. Bard is clearly inferior to GPT-4 and Google has no shortage of GPUs or electricity (or data).

> Bard is clearly inferior to GPT-4

Have you tried it recently? It got a lot better, and I like it more for research tasks. (GPT4 + Bing are alright)

It’s actually insanely good now, especially with images

$100m is relatively cheap for something at the scale and sophistication of gpt-4. I think money is not the issue-it’s all the little things that had to be done right as Altman said in an interview.

So one can burn $100m and not have gpt4. Google.

They're working on Gemini which is supposed to blow gpt-4 out of the water. I believe it when I prompt it.

We have seen this happen with Google+. Only Google pawns trying to convince people to use it.

H’s for the RLHF too?

I guess that is easy to solve by throwing money

I believe it's it's humans (for RLHF and other things) and data. If it were just power and chips, the current front-runner would be a corporate behemoth or a state, not OpenAI. (They might become a corporate behemoth soon enough but they aren't so far.)

At this point I'm pretty convinced that no new gigacorp will be emerge out of this generation of AI (LLM and vision models). OpenAI will have to continuously bring their A-Game to stay 6 months ahead of everyone else. There is durable moat that they could hide behind in sight as of now. Of course they would like to construct a regulatory moat but it doesn't seem like there is a clear majority for this.

I'm pretty sure and happy that this will probably turn out as a large consumer windfall. Actually comes right in time to counteract the strong inflationary forces that come with de-globalization and declining efficiency due to increasingly basing the economy on gov spending.

I'd say expertise and compute. With enough compute I can certainly build the largest LLM. Do I have the expertise to build the BEST largest LLM? No.

Easy. Human feedback and training. The open secret is that just the AI “revolution” is built upon uncompensated or barely-compensated human labor. Not just everyone uploading their stuff to the internet for 30 years but also this:

https://www.forbes.com/sites/kenrickcai/2023/04/11/how-alexa...

Most people on HN have no idea about that

https://futurism.com/the-byte/ai-magic-spy-on-you-experts-wa...

And especially OpenAI:

https://time.com/6247678/openai-chatgpt-kenya-workers/

That’s the dirty secret of why ChatGPT 4 is better. But they’ll tell you it has to do with chaining ChatGPT 3’s together, more fine tuning etc.

They go to these poor countries and recruit people to work on training the AI, scan their eyeballs for crypto and whatever else. And then privatize the profits from all this.

But it’s nothing new, capitalists have always exploited labor… and people make their choices.

At least it isn’t ivory or blood diamonds

https://www.reuters.com/technology/kenya-panel-urges-shutdow...

And then they added their guardrails and made sure only positive, happy answers would be give

I'm wondering if the interconnect between GPUs in this HPC which seems like a more traditional supercomputer good enough for LLM training?

In my opinion They won't outperform the best LLMs for one problem: Quality of data.

A hidden secret that everyone knows is that good LLMs use copyrighted data like books-3. OpenAI itself cites a dataset from books and does not go into detail in its first papers.

Is bigger always better? Surely "bigger is better" is at odds with "garbage in, garbage out". But I am not an LLM modeler. IOW can LLMs be improved by improving the quality and/or even *limiting the size of their "corpus"?

Could we instead use, as corpus, a select body of knowledge primarily based on Western sources such as those texts a student might read as part of an excellent liberal arts education?

Further, would it not be possible to limit the corpus text (and consequently later, queries) to grammatically correct language? That is, must we include all the "errors" of common speech in a corpus, such as occur especially in fiction? Indeed, could we not exclude fictional works altogether, esp. if the model is to be used for scientific work?

I'd be curious to see an LLM trained on "The Great Books" plus a college course plan of textbooks. But again, I am not an LLM modeler.

Anyone experimenting with such "Small Language Models(SLM)"?

The TinyStories paper[0] explores some of the potential of "small language models." I think you don't need as much data if you have really high quality data (really high quality); however, having a lot of data seems to be able to compensate for widely varying quality.

[0] https://ar5iv.labs.arxiv.org/html/2305.07759

Thank you for the reference!

But as long as there is low quality data might not that data "pollute" the process (whatever that may be)? i.e., "GIGO".

Is "low quality data" possibly necessary for a properly-working LLM?!

[Aside: why the downvote?]

Speaking of "secrets":

We didn't create an open-source project for the atomic bomb. Quite the contrary: it was one of the best-kept secrets for years.

Sooooo...

if LLMs are as revolutionary as some claim, shouldn't there be significant effort to keep new advances and developments secret and lead competitors astray? That is, if I knew something that would provide a 10- or 100-fold improvement in LLMs, my tendency would be to profit/benefit from that rather than publish it. And to mislead competitors? Has LLM technology development gone "underground" yet?

And what is the role of the 3-letter agencies and the government in this radical development?

Or is LLM technology all unicorns with rainbow farts where the entire world benefits and the creators get only thanks? Somehow I don't believe so.

Will it actually be open? As in weights available?

Good for them, Europe should compete against US, China, etc.

It must be nerve-wracking being in charge of huge multiple month training cycles. In the 1980s, we built our own hardware to speed up back propagation learning and recall and runs could still take a day or two. It was always so disappointing when a long training run went bad. I can imagine how tense the responsibility must be to manage these huge training cycles.

Are there no incremental ways to verify that it's working?

Yes, you can. After every training batch, the network's loss function is updated and you can track it over time to verify that it's going down. Sometimes, the loss can get stuck on a plateau, or even go up, and it's not clear whether it's worthwhile to keep training or not.

Traditionally, training runs can "explode" and fail, but there are methods to incrementally back them up and resume when that happens, see https://www.mosaicml.com/blog/mpt-7b

Of course, you can measure loss on a dev set, etc. However, LLMs are so complex that until they are done training and many people evaluate them, there will be misses.

What political lean will a LLM have... we've had some really bad politics come out of Europe throughout history.

Is this the beginning of a LLLM? (Lenin LLM)

It will for sure be well versed in government overreach

Yeah, everyone claims sota, but the funniest ones are the ones claiming it before training even starts

Also, the focus on "trustworthy" seems problematic. Beyond the obvious who is defining what can be trusted, trustworthy models are significantly harder to work with.

Meh. PR overreach hoping for a Euro vanity project. University of Turku isnt a "hotbed" or "powerhouse" of LLM experts

It seems quite a well positioned university. This dismissal seems unwaranted.

The TurkuNLP team does have the best previous-generation language model for Finnish (FinBERT). The current version of ChatGPT speaks passable Finnish, although definitely not at the level it speaks English. None of the open-source models I've tried come close to ChatGPT performance.

If they can gather a good multilingual dataset including the smaller European languages, and burn enough money on the compute, they could create a useful model for some specifically European use cases.

Largeness of the LLM shouldn’t be the goal.

More data, more parameters, more compute all result in a better model per "Scaling Laws for Neural Language Models"

https://browse.arxiv.org/pdf/2001.08361v1.pdf

Largeness is a valid goal.

Also: costs more for inference, uses more energy, less practical for running locally, fewer use cases as a result. Especially for an open model.

Being on Github / HuggingFace but needing to be on a AWS or Nvidia wait list to get the resources to run it is not great.

In an unlimited energy and chip world I would agree just make em bigger.

I guess going bigger has a greater chance of success in being SOTA than looking at architectures. So I get people don’t want to gamble.

rebuttal: compute optimality matters https://arxiv.org/pdf/2203.15556.pdf

Watched a debate on major media channels where people were arguing that pictures of killed children were AI pictures (and the opposite side refusing it)

What kind of a world we have devolved into, where technological progress makes us less human, on average.

I am saddened by the changes brought about by AI, be it generative art or LLMs, where, because of it, on a global scale, no one knows what truth is, even in pictures and videos.

And we have the perfect tinderbox of siloed wells on social media, with tools to generate authentic looking content that can be disseminated at essentially 0 cost, automated at 0 effort and be used to indoctrinate people at 0 consequences from governments.

It does not matter if people are caught after the fact, the mind that has been indoctrinated, cannot be reverted with the same ease.

And we are generating a bunch of children now, who will be adults later, who will essentially reject reality, reject the need for empathy because what they see may or may not be the truth.

It used to be that 10 years before, if one saw the pictures of carnage or destruction or dead children, a spark of empathy may be generated, enough to ignite change, no matter how small. In the near future, that may not be the case because that photo or video or content has the probability of being AI generated and that spark of empathy will die without a chance to ignite change.

And now we have billions of dollars being invested by the very governments that will feel the effects of these short sighted decisions to unleash technology seldom understood and have no clue as to its long term effects.

Reply on news.ycombinator.com