I would expect a language model to pull up different person names for each time that one was called for. For a person name to be consistently used through several paragraphs it is not enough to rely on word co-occurrence.
If I had to produce a text like this I would simply take an existing text and replace randomly chosen words with other similar words (as hinted by amvalo). Similar as in - words that tend to occur in similar contexts. So John->Bob throughout entire text. But that would not be a language model product anymore, and where is fun in that?
I should set aside some time to read this paper.
With a temperature of 0.7/1.0, that's enough for sufficiently random text I suppose. (the raw, uncurated generated text using the smaller model is a bit more random: https://raw.githubusercontent.com/openai/gpt-2/master/gpt2-s...)
For example, the generated text about the Civil War mentions that Thomas Jefferson Randolph [0] was named after his grandfather, the president. But is the wording mostly influenced by articles talking about that specific fact, or does it draw from more general examples of someone being named after their grandfather?
From a safety perspective, it would be useful to see what prompt a piece of text might have been generated with...