nujan_dev

Large models don't just tolerate noisy or nominally "low-quality" web data but this paper suggests that they benefit from the distributional entropy in it. Over-curated datasets artificially compress the representation manifold, starving the model of the edge cases needed for robust generalization.

A little misleading. The title makes it seem like the robot learnt to write these unknown langauges but it looks like it was shown an image and was able to reproduce what was on the image, even though some of the writing was not in its training domain. Pre-LLMs so still pretty neat, just less impressive than the title suggests.