Accuracy is not really the right metric. In my opinion, there would be a lot more value in looking at the sensitivity and specificity of these classifications by humans. They are on that track with the logistic modeling and odds ratio inherently but I think centering the overall accuracy is wrong headed. Their logistic model only looks at what is influencing part of this - perceived and actually ai generated text - separating those features from accuracy to a large extent. I think starting with both the AI Overall, the paper conflates (to use medical testing jargon) 'the test and the disease'
Sensitivity - the accuracy of correctly identifying AI generated text (i.e., your True Positives/Disease Positives)
Specificity - the accuracy of correctly identifying non-AI generated text (i.e., your True Negatives/Disease Negatives)
these are fundamentally different things and are much more explanatory in terms of how humans are evaluating these text samples. It also provides a longer path to understanding how context affects these decisions as well as where people's biases are.
In epidemiology, you rarely prioritize overall accuracy, you typically prioritize sensitivity and specificity because they are much less affected by prevalence. six months ago, I could have probably gotten a high overall accuracy, and a high specificity but low sensitivity, by just blanket assuming text is human written. If the opposite is true - and I just blanket classify everything as AI generated, I can have a high sensitivity and a low specificity. In both cases, the overall accuracy is mediated by the prevalence of the thing itself more than the test. The prevalence of the AI-generate text is rapidly changing which makes any evaluation of the overall accuracy tenuous at best. Context, and implications, matter deeply in prioritization for classification testing.
To use an analogy - compare testing for a terminal untreatable noncommunicable disease to a highly infectious but treatable one. In the former, I would much prefer a false negative to a false positive - there is time for exploration, no risk to others, the outcome is not in doubt if you are wrong, and I don't want to induce unnecessary fear or trauma. For a communicable disease - a false negative is dangerous because it can give people confidence that they can be around others safely, but in doing so that false negative causes risk of harm, meanwhile a false positive has minimal long term negative impact on the person compared to the population risk.