The Pile: An 800GB dataset of diverse text for language modeling (2020)

Personally, I'm not worried. It would be a damn shame if academics come under fire merely for trying to operate on the cutting edge of science. None of us were trying to make money; we just wanted to make something interesting.

> I'd also be keen to hear how your challenge against the DMCA on sharing LLaMA's weights goes?

Thanks! I think we might be putting up a website for it soon, if only to explain ourselves. In the meantime – I hate this phrase, since I don't want followers – the only way to keep informed is to follow my Twitter, and perhaps keep an eye on my HN comments.

You'll probably hear about it either way though, since it's a groundbreaking case. No one has tested the copyrightability of ML models before.

Ultimately what convinced me to proceed is that there are immense forces pressuring ML models to become SaaS companies. It's very difficult to offer an ML model for extended periods without being a company. E.g. https://6b.eleuther.ai/ is down. Eleuther failing illustrates just how hard it is –– we were all working as hard as we could to design something that would last a long time, and a long time turned out to be two short years. Contrast that with other kinds of hacking (e.g. webdev, gamedev, hardware...) where the end result lasts basically forever.

So if ML models aren't copyrightable, I think it'll hurt companies a lot more than individuals. In fact the goal is the other way around: to protect individuals. All I did was publish Facebook's own GPL download script to github, and it got DMCA'd. If we don't push back on that kind of behavior now, companies will get used to the idea that they control "their" model –– even when their model is anything but theirs.

At some point this stuff will have to get tested by making its way up the appeals stack in the US, and IMO there is only a minuscule chance that will result in Google, MS, and Meta getting slapped with anything more than a token fine (my bet is it won't even be that), let alone paying every person who ever wrote anything that was used in these datasets for copyright violations, which would basically be everyone.

I think there's a strong argument for a Fair Use defense, given the size of the models versus the size of the training sets, as well as the gulf in intended use: an AI model doesn't compete with e.g. a book. Obviously we'll have to see if play out in court to find out.

From a scientific point of view this is very problematic because few safeguards exist that guarantee that the dataset is not tampered with (as is the case if you'd upload it to Zenodo, which providea some guarantee of immutability).

How about trying to upload the Pile to Zenodo? Only half-joking :D

I’ll read over your essay and give it some thought. There are a bunch of subtle aspects to consider; I’ve been thinking it over for about four months now and still haven’t covered all the territory yet.

It feels like this may be one of the most important decisions going forward — both from an intellectual property point of view, and an individual rights perspective. E.g. you mention that it’s civil disobedience to share the weights, but it feels like if someone is claiming to do open science (LLaMA), sharing the research materials is the minimum requirement. Plus look how it’s benefited them; they’ve captured most of the open source LLM mindshare. So it seems likely that this will lead to more open source work in the long run, not less.

Feel free to chat! You can DM me on Twitter or email me. I’ve been in the hospital with my wife for 7 weeks, with two to go, so I’ve been a bit less responsive than I usually am.

We will see that anyway. All the code I work on commercially is copyrighted and yet a trade secret. Existence of copyright (with the exception of copyleft, but that's subversion) didn't help software to be open sourced.

IMHO allowing models to be copyrighted is basically 18th century enclosures again.

Besides, it's already pretty unambiguous that weights are not copyrightable: they're a result of a mechanical process. The only original creative input that goes in to the weights is the unfathomable amounts of content scraped from other sources that aren't the authorship of the models. The objective of the gradient descent is simply minimizing loss on the training data.

Facebook doesn't own the llama model weights any more than the Bridgeman Art Library practically owns the paintings of European masters because they made quality scans of them. ( https://en.wikipedia.org/wiki/Bridgeman_Art_Library_v._Corel.... ), or any more than Rural Telephone owns the phone directory ( https://en.wikipedia.org/wiki/Feist_Publications,_Inc.,_v._R.... ).

Trying to make model weights copyrightable is going uphill, and I don't see how you get there without first establishing that the these LLM are unlawful derivatives of a countless number of copyrighted works along the way. Doing so would probably create a immediate monopoly for legally created LLMs for the hand full of corporations with quasi-monopoly content hosting services (facebook, google, etc) that can (and/or already have) stuffed licensing into their terms of use.

Do you want a cyberpunk dystopia? I think creating an AI monopoly is how you get a cyberpunk dystopia -- and the two ways we end up with one is either outright restrictions on private development of AI like some have been lobbying for and the other is the extension of copyright so that only a few entities can get access to enough of other people's data at a low enough cost to train them.

Does anybody have a link to a relevant discussion here? I would like to read about the creative process that goes into defining model weights, and how it differs from the mechanical output of running the training algorithm.

Furthermore, I think you are exaggerating the memory issue of multilingual models significantly. Especially for languages using the same (Latin) script, the additional characters to care about are very few. Also a significant part of the vocabulary and language fall into a few buckets, so training a joint model makes all the sense in the world - much like an Italian native speaker could likely study a scientific text in Spanish and understand its content, even without speaking the language.

The memory impact comes mostly from having bigger embedding layers that have to account for vocabulary in many languages (the most problematic case being Chinese and Japanese, with their huge set of tokens). But even there, the largest vocabularies in use are maybe of size 100k (vs. about 30k for English-only), with a hidden dimension of 4k that makes for a total of 400M parameters. It's a lot, but a drop in the ocean of 100B+ parameters (or 1T+ for GPT4) we're seeing today.

P.S. Answering to GP, I think the Pile is English only, though - or at least, models on HuggingFace trained on the Pile, like the various Pythia models, are tagged as English only.

Also, there are lots of (poorly) machine translated websites in Polish... So any dataset that contains web crawl will have precisely what I'd prefer not to have.

Ideally I'd see either the national gov, or the EU to invest money into creation of more high quality datasets in all EU languages.

So when I point out the failures of for example chatgpt in my language I do so while being amazed it can generate and understand Polish at all.

Also regarding multilingual model size being larger. I never heard this before, but it seems logical. I have heard models gain extra performance on English tasks when they are trained on other languages too so there is a benefit to adding multilingual datasets.

Kinda of sad to have missed NeurIPS dataset track deadline and ACL, but I know that anything close to this in scope is a slam-dunk accept at the argument mining workshop