Pretraining on the Test Set Is All You Need
arxiv.org
arxiv.org
What's left, after you use, say, "half" of The Entire Internet for training? What's left that is definitely not the same as the half you used? How can you be sure, when you're working with terabytes or petabytes?
Credible benchmarking is going to have to become a field in itself.
Maybe it needs to be obfuscated, or all the questions should be obscene so they don't pass the content filtering.
The problem that this "paper" is making fun of is that any model can score very well on these tests when the content of the tests themselves is used in the training of the models. Here is one example of this happening with a 13B model that scored way too high. [2]
In reality, when reviewing test scores, it's difficult to discern whether a model has been pre-trained on some specific content that is related to the test, or if the model has been trained in a way that produces novel emergent abilities not seen in other models. [3] This means that LeaderBoards and test scores for LLMs provide a low-confidence metric for what the propose to be testing.
The paper's title is itself a continuation of the phenomenon where papers are named after the landmark paper "Attention Is All You Need". [4] A good example of this is the "Textbooks Are All You Need" [5] paper that propsed pre-training in an incremental way, similar to how humans learn, produces novel results.
1. https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
2. https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
3. https://arxiv.org/abs/2206.07682
Full disclosure: I have an AI side project that also uses “is all you need” in the title.
The implementation is just enough to run very small workloads:
* Full GPT2 small (124M parameters) model including byte pair encoding, embeddings, multi-headed attention, and multi-layer perceptron stages
* Inference/forward pass only (no training)
* Context is limited to 10 tokens in length
* 10 characters per word limit
* Zero temperature output only
This sheet is very big. Unfortunately, it is not unusual for Excel to lock up while using this spreadsheet. It is highly recommended to use the manual calculation mode in Excel.
(apparently, a lot of things are "all you need" !)
(or, in other words, "Is All You Need" Considered Harmful)
Imagining it was not clear enough, the author decided to name the model something 'pronounced “fictional”', and shows how it can answer questions that weren't even asked yet with perfect accuracy right on the abstract.
Accept that as an exercise of familiarizing yourself with the area's language. If you are interested on it, it's not lost time.
If it takes you reading the whole paper up to the point where the author explains the joke to realize that the paper tagged satire is satire, I mean, I'm not sure paper-reading is for you. There are unmarked papers out there that aren't even satirical, just wrong!
I guess there are cases where training on a tests set is reasonable, but why use a LLM in that case?
I find this annoying, and not funny, honestly.
Why are satire papers mixed in with real ones on arxiv.org? Obviously, I'm not a research scholar and don't know how the whole process works, but as someone who likes to know whether they are dealing with fact or fiction from the start, this was a waste of time.
> Comments: 3 pages, satire
Tangentially, it's worth pointing out that you are gatekeeping read access to a resource because of a write access problem. If you don't want open access then go back to Elsevier.
I have no problem with open access, and would rather journals were required to be open access. But reading brand new results that haven't had any peer review without familiarity with the field is putting the cart before the horse. I certainly don't learn a new subject by reading the hottest papers on arxiv.