That ignores random sample problems and the simplistic statistical analysis…so yeah it unfortunately does.
It doesn’t prove they are wrong, but it’s not credible to say they are right
The problem is that unless the data are credible and reliable, any interpretation of that data is meaningless. For or against one’s hypotheses doesn’t matter if the data is junk. If the data is junk you can’t move past that, it’s the end of the conversation because the data has no meaning.
Perhaps by "random sample problems" you mean that the study is not representative of all of humanity? If so we can still take the paper as evaluating these 415 humans who speak English against the two models. If as you say, the workers are actually just using LLMs then this implies there is some LLM that your average MTurk worker has access to that out-performs GPT 4 and GPT 4V. That seems *extremely* unlikely to say the least.
There is no need for any complex statistical analysis here since the question is simply comparing the scores on a test. It's a simple difference in means. Arguably, the main place that could benefit from additional statistical procedures would be weighting the sample to be representative of a target population, but that in no way affects the results of the study at hand.