> unfortunately the tokenizer was trained on this subreddit
Do you have a source for that or was it just an assumption?
Do you have a source for that or was it just an assumption?
Also, that Reddit is frequently used to train LLMs is widely known. It's an unusually clean source of conversational text because you can slice threads (i.e. pick a root comment, then pick a child, then a child of the child etc and then concatenate the results), and you'll get a coherent conversation. There are relatively few places on the internet where that is true. For example most phpBB forums conflate many different conversations into single threads, with ad-hoc quoting being used to disambiguate which post is replying to which. That makes it a lot harder to generate sample conversations from.
Imageboards.
DailyMail.
Slashdot.
Even a somethingawful dump would have been superior.
The Daily Mail (the newspaper) has been used for training LLMs in the past, yes. I don't know if it still is.
listen, some of the niche corners of that world aren't so bad, but it ain't the place to be training AI to do something, unless that something is a hate crime