FineWeb: 15T tokens of the finest data the web has to offer

huggingface.co

12 points·tosh··3 comments

Lovely! Right there in the first page of example strings:

> Huge selection of exclusive animal sex videos is what makes 3AnimalSexTube the best animal porn tube site. The site contains 100% user-uploaded free porn. Feel free to send us your feedback or question. Attention! 3Animalsextube.com contains materials of erotic character and it is intended for viewing by an adult audience. Parents, protect the minor children from pornography viewing in a network the Internet ICRA or ASACP This web site is for adults only. All Rights Reserved.

" by carefully adding additional filtering steps, we managed to push the performance of FineWeb well above that of the original RefinedWeb, and models trained on our dataset also outperform models trained on other commonly used high quality web datasets (like C4, Dolma-v1.6, The Pile, SlimPajama) on our aggregate group of benchmark tasks.

That said, we think there is still room for additional filtering and improvement and intend to continue exploring how to improve the dataset quality in coming versions of FineWeb. "

I think future advances in model quality are more likely to come from better filtering to decrease the amount of bad training data, rather than training on an ever growing share of the entire internet.

I don’t think that’s bad to be in the raw training data per-se. Such content will be RLHF’ed out. Some users will certainly ask it for such tasks and it will somewhat deliver

Reply on news.ycombinator.com