The study also doesn't really discuss the interesting question of whether ChatGPT's answers for either the Democrat or Republican are accurate, or just GPT trying to guess responses that fit the correct quadrants (possibly based on meta discussion of the test). Although given that the party labels apply to pretty broad ranges of politicians and voters, I'm not sure there is such thing as an "accurate" answer for the position of "a Republican" on Political Compass, especially when it comes to areas like use of the military overseas where the Republican party is both traditionally the most interventionist and presently the most anti interventionist. The Political Compass authors, notably, put candidates from both major US parties in the authoritarian right quadrant which is quite different from GPT's (though I find it hard to believe the academics' scoring of politicians is an accurate reflection of how the politicians would take the test either)
Plus of course there's also the question of what "politically neutral" is... and the idea that a "neutral" LLM is one which agrees with Jair Bolsonaro half the time on forced choice questions (perhaps if it disagrees with the proposition that people can't feel naturally homosexual it should balance it out by seeing significant drawbacks to democracy?!) is questionable to say the least
I do think that the questions need some work, but there’s a tension between that and maintaining consistency over time so that results can be compared over time.
There are a number of problems here, but the biggest one, is individuals are very commonly not internally consistent on their political views. What is GPTs inconsistency in actions/replies compared to that of varies views on the political spectrum is far more interesting to me.
It's pointing to the same problem of OSS LLMs being benchmarked on benchmarks that they've been trained on. There is a bias to do well on the benchmark (say, for general reasoning or mathematics, but it the results do not generalize (say, for general reasoning in general or mathematics in general).
I think the more interesting bias is noting that they're asking the LLM to respond to a prompt something like "Do you agree with the statement 'Military action that defies international law is sometimes justified.'" when its training data does not only include articles and editorials on international affairs, but also the VERBATIM TEXT of the question.
Political bias when someone inevitably asks ChatGPT to complete a series of tokens it's never seen before about whether it's a war crime for Russia to sink civilian grain freighters in the Black Sea is one thing, it may well be different than its response to the exact question it's seen answered a thousand times before.