> One popular dataset, FineWeb, is about 11.25 trillion words long3, which, if you were reading at about 250 words per minute, would take you over 85 thousand years to read. It’s just not possible for any single human (or even a team of humans) to have read everything that an LLM has read during training.

Do you have to read everything in a dataset with your own eyes to make sense of it? This would make any attempt to address bias in the dataset impossible, and I think it's not, so there should be other ways to make sense of the dataset distribution without having to read it yourself.