>If anything, this release makes me excited for the quality of smaller models going forward.
While there's going to be more that can be milked out of them, we are already clearly fairly deep in the diminishing returns portion on small models.
While there's going to be more that can be milked out of them, we are already clearly fairly deep in the diminishing returns portion on small models.
Llama - 1.4 Trillion tokens - feb. 2023,
Llama 2 - 2 Trillion tokens - july. 2023,
Mistral - 8 Trillion tokens - sep. 2023 (this was the first big impressive leap where local models really became useful for chat)
Llama 3 - 15 Trillion tokens
So we went 1.4x, to 4x, to 2x. I wonder if there's even unused data out there?
Haven't tried the new model enough to see how much better it is than Mistral, that will be the real SOTA test for now.
Also the redpyjama v2 dataset has 30T tokens and is based on common crawl. Now I don’t know too much about common crawl but I doubt it has in it all published scientific books in all the different languages as these are often not freely crawlable. I remember when I studied physics there were at least 20 different 200-800 page long books on particle physics in German alone in our campus library. That must amount to 5million token by itself, from just one niche of physics. The Hamburg public library hosts about 5 million books and 90 million scientific articles mostly in English and German. If the average length of a scientific article is 6000 tokens and the average book about 100000 then that alone is already 1 trillion token. I bet there are significantly larger libraries and this is before even crawling the internet and looking at other languages or even generating training data.
E.g. the Norwegian National Library has somewhere between 3x and 10x as many tokens in Norwegian newspapers as in books (at one point I think GPT3 breakdown of training data by language surfaced, and the Norwegian data was a tiny fraction of what was available in the national library, even before trying to estimate online/digital content).
While I'm sure there's overlap [1] between the languages, a lot of it will help translation, and I think even for smaller languages the ratio of local content seems to dwarf translations. E.g. the "bestsellers" from English, French, and German all get translated to Norwegian, but most of the "long tail" content is local.
[1] I was tickled to a find one of my uncles represented in Deutsche Nationalbibliothek; he was a professor in statistics, so it was a translation of some of his research
One day someone is going to train on the contents of DM's/private conversations/emails. There has to be 50x or more the quantity of that compared to public text.
I suspect they'll do it via some 'prove-ably private training' regime, and therefore be able to claim it isn't a privacy violation.
They also have youtube, which almost certainly has enough good data to train a powerful model on it's own, but also seems daunting to clean up first.
I think you're massively underestimating the amount of data out there. The challenge is how to access and categorize that data.
Every usenet message, forum post from old school bbs to php forums to modern javascript abomination and closed discord boards, every email, every text message, every irc message, ...
Those are probably a pain point to access due to rules and regulation and yada yada, but that alone dwarfs the 15 trillions, and you've not even started on actual quality content.
(not saying these would specifically be good for llm, just answering to your "unused data" assessment)
One thing though is books/content/media from other language spheres though that could probably at least 10x the size of the data, and as far as i know translation starts to work rather well in these larger models so it would probably just plug right into the knowledgegraph for all languages?
Similarly I was looking for something from the British Library at one point and it was behind a paywall (a copying fee).
I have no idea how to even start to estimate how much data is locked down like that, and it's harder yet to try to figure out which parts of that it'd be possible to negotiate access to for various players, and what they can circumvent (e.g. say by buying book collections and the like - OpenAI is large enough by market cap it could afford to buy some of the largest extant publishers, for example, if they thought it gave them sufficient benefits).
In other words if LLM's could somehow bridge that gap through both regions and time i'm pretty sure something magical could happen, different than the already a bit tired and conformist "echo chamber" like quality to LLM's mostly trained on reddit, corporate speak, and anglo pop culture, or even just western thought in general.
That said, English and Norwegian are pretty close. How well it will handle languages with more significant differences without larger amounts of tokens is another matter. Even for pretty small language groups there ought to be enough, though.