Hacker News Reader

Best sentence in the article: "It also happens to be trained on a large chunk of Reddit, since the author decided that this was undeniably the perfect location to obtain high quality, impeccable prose."

View on HN

While snappy, this sentence is misleading. The model is trained on webpages whose links have received at least 3 karma on reddit, not on reddit comments themselves [0]. The motivation for this choice is not to find impeccable prose, but rather to establish a lower bound on quality for the dataset, and avoid training the model on too much of the spam/nonsense text that is frequent in e.g. CommonCrawl. Also, the "author" is not a single person, but rather 6+ people [1].

[0] https://openai.com/blog/better-language-models/ [1] https://d4mucfpksywv.cloudfront.net/better-language-models/l...

Reply on news.ycombinator.com