I don’t think text is sufficiently unconstrained. It is trivial to get GPT to output “Hi, how are you?” but of course it’s impossible to determine if that common turn of phrase is human or machine generated.

There is too much a correlation between the position of tokens and correct meaning. In contrast, in an image, it’s very possible to have pixels shuffle around and not change the look. There, it is more likely that we can determine that the output is statistically more likely to be from a particular model.

Even then, you don't know the prompt.