This is most apparent with older models like GPT-2, and more generally base ("text") models like Davinci. They are essentially only trained to predict the next word, based on what they've seen on the internet.

So they're a really fun way to try and "sample" the internet, by seeing the kind of outputs they produce.

--

A really fun thing I did with GPT-2 with a friend back when it came out, is we fine tuned the model on our WhatsApp conversation together.

(The model was small enough to do this on mid range GPU, in a few hours.)

So GPT-2 learned to simulate chats between us.

It was hilarious, but also weirdly illuminating.

My friend mostly talked about art (true), and didn't say much (true), and I rambled about how I finally figured out the One Weird Trick that's finally gonna turn my life around (...also true)...

The text itself was largely incoherent, but the "feel" (statistical properties?) were spot on.