I think this is an important problem. With logistic regression or deep learning, at least one can compare (out of sample) calibration curves or discrimination measures. With a language model, what can we do?
perplexity score against a corpus such as wikipedia? Basically how well the model predicts the next word.
This is a good start, but given the breadth of applications this would hardly give us enough to compare, as the goal of these models isn't to simply recite Wikipedia articles. What about language translation? Content summarization? Code generation? Turing test performance?
Both models were trained on Wikipedia, so that's a particularly bad choice. But yes, in practice this is what people tend to do. Take results with a very large grain of salt though, as the domain of the prompts you feed it make a huge difference.
yes, see GLUE or superGLUE benchmarks. It assumes the answers have not been scraped and included in the dataset though.