Hacker News Reader

I would think if it were AI generated it would use common words. You know, like from its training dataset...

Tokens consist text both larger and smaller than whole words.

The tokens that make up rare words would still likely be just as rare as the words.

And yet GPT-4 has no problem crafting new words from whole cloth. If you're making new words rare tokens are useful and the model knows this.