"Unlike in OpenAI's blogpost, I did absolutely nothing to manually cherry pick the quality here. Some examples are good, most are bad. Almost all outputs decay in quality as the post gets longer. I wanted to keep the raw outputs to give an accurate feel of the strengths and failure cases of the model. The truth is that using a GPT2 type model to create text you want is more an art than a science and can be very finicky."
Then of course there is the fact that the network can generate text a lot faster than a human can read it. Think for a moment how many distinct passages these systems can generate -possibly even infinite- and how few of them a single person can realistically hope to read. It's very hard to know if the few passages one ends up reading are representative of the output of the network, or not.
It'd be even harder to compare two systems just by eyballing their output side-by-side.
Given the large difference in quality, it only takes a few comparisons. Again, just look at the samples. Read a few. Do they really look the same in coherency and quality and realism?
Looking at two texts side-by-side while knowing which is which may well create an illusion of striking differences, which however are not really there. This is why concrete metrics are necessary- because human judgement is biased and inconsistent.
But, like you say- there are no good metrics. In fact, the only thing we have to go by when discussing Open AI's model in the first place is opinions and convictions. Not least the opinion of OpenAi that their model generates text of unprecedented coherence, which itself is not based on anything else than eyballing.