A closer look at BookCorpus, a key dataset in machine learning | Hacker News Reader

A closer look at BookCorpus, a key dataset in machine learning

towardsdatascience.com

122 pointsKaibeezy32 comments

This is really a fantastic analysis of a dataset, and it's something that should be a mandatory form of smokescreen before proceeding with actual model training, in every organization or research group. Whether you are using public datasets, buying 3rd party data, doing in-house data collection and annotation, or paying someone to do it for you, you must check for class imbalance and over/under representation within your data - inevitably human biases will creep in. Ultimately you have to evaluate whether this data distribution is compatible with your target data distribution your model will be applied on in production. Doing this post hoc is a real pain.

I agree that this is extremely valuable, but it's worth flagging that it's harder to reason about the impacts of class imbalance for generative models than e.g. classifiers. For example, should we think about genre imbalance per novel, per token, or on some more complex basis? Which genres are most relevant to a target distribution of chatbot queries?

This isn't to suggest that organizations shouldn't invest in actively understanding their training data, but that post-hoc bias analysis is going to be a critical component of evaluation for the foreseeable future.

The Wikipedia article on BooksCorpus raises an important point about how researchers used the word "unpublished" to describe the books in this corpus -- this word appears in both the original Aligning Books paper, as well as OpenAI's papers, which don't even bother to acknowledge SmashWords. The books aren't "unpublished" -- SmashWords is a self-publishing platform! Whether deliberately or not, the word choice diminishes the human effort that was appropriated by the researchers to train the models.

OP here. I had a lot of difficulty finding info on Books1 and Books2, even on HN. If there's a better source of info, please link or post.

What's the value of these scant few thousand unpublished romance and fantasy novels in the context of the rest of the corpus -- vast scrapings, all of Wikipedia, etc.? A sample of how people write? Why aren't more public domain works included?

> What's the value of these

The Pile (the 800GB dataset by Eluther AI) contains BookCorpus2, along with two much larger datasets of books (and a whole lot of not-book stuff). From their paper [0] the reasoning for the book datasets is that they are "are invaluable for long-range context modeling research and coherent storytelling". The reasoning for including BookCorpus2 next to Books3 and Project Gutenberg boils down to "no significant overlap with the other datasets, and others use it".

In general books are a great source of extremely high quality long-form content. They are longer than most content found on the web, and are generally of high quality, having gone through many revision rounds between editor and author. Just that both of these aren't really true of BookCorpus. Even a dump of highly rated stories from fanfiction.org might be better.

0: https://arxiv.org/pdf/2101.00027.pdf

Are these primarily fiction books, or a mix of fiction and non-fiction?

The books in the books3 collection aren't categorized. The source, however, currently is at a ratio of 2:1 nonfiction to fiction, and from what I've seen, whoever created the books3 archive simply attempted to gather all the EPubs they could, with their only criteria being availability.

I wonder how much better an LLM could be given even better training data.

For example, the total number of tokens contained in the physical and digital collections of a moderately-sized university library is (probably) equal to or on par with the size of the training data for GPT 3.5.

What would happen if you could train just on that? I know we're using huge training sets, but how much of it is just junk from the internet?

(There should be some representative junk in the dataset, but nowhere near the majority.)

Isn't this what "tiny stories" / Phi LLM are doing? https://arxiv.org/abs/2306.11644

https://arxiv.org/abs/2306.11644 is along these lines.

I see many courses and papers base their work off BookCorpus. So this should be somewhat significant. What are the 'places' that these concerns can be highlighted to the wider machine learning community? Not everyone will be here. Is there a forum or a 'reddit' or a conference where machine learning people regularly visit?

The cynical me is thinking that because this is inconvenient news and would require rework, a lot of people would prefer to suppress or ignore the author's findings (assuming true).

This is from 21, not really news, and the paper version on arxiv and published at NeurIPS have quite a few citations. No one's suppressing this, people that don't reflect on their datasets or how they use them just either don't care or fail to acknowledge they're actual issues.

IANA AI developer but have been looking into this in detail recently for other purposes. I was puzzled at the lack of info about "books" and when searching for detail (in what I believe was a reasonably diligent manner) found a very surprisingly small amount of it. I assumed there would be more knowledge and did ask for it here. So now I will go look up those papers to get a better sense of things. Thank you for the tip.

I note neither this paper nor any discussion of "BookCorpus" or even "book corpus" has appeared on HN previously.

Addressing "Documentation Debt" in Machine Learning Research: A Retrospective Datasheet for BookCorpus, 2021, Jack Bandy and Nicholas Vincent

https://arxiv.org/pdf/2105.05241.pdf?

The correct title is “Dirty Secrets of BookCorpus, a Key Dataset in Machine Learning”.

HN sometimes changes titles to make them less lurid, clickbait-y, etc. Whether that's a bot or not ¯\_(ツ)_/¯

In this case, the modified title utilizes the more descriptive language of the article's subtitle. Editors edit.

I didn’t realize, I thought it was an ironclad rule to use the original title. Thank you.

I regret reading about half of that article and suggest you save the precious moments of your life that reading it would take and do something that is valuable out interesting instead.

Reply on news.ycombinator.com