All opinions are biased. It is a tautology. I guess my biased question is to do with cost. I’ve not been able to run any models of the caliber that could pass a medical test on my local machine. What kind of hardware do I need to own or to lease in the cloud to write code that doesn’t arbitrarily include comments without comment initializing tokens, and how much will the whole pipeline run me? Is that already cheaper than my salary? Some 25% of code at Google is now written by AI, but all of that code is still reviewed by humans. Is it worth it if you still have to pay for both? Quite possibly, for some of the huge companies, but most devs are working in smaller shops: is it still a reasonable expense for them?

Is passing a formulaic test a high bar? Lot of people pass these tests with training. Taking the lawyer example, does model that passed test also be able to correctly write all documentation for a case from start to end? Without hallucinating any non existing cases or references?

Is it possible the people administering these tests are less than honest about the outcome? Perhaps they have a vested interest in the results they've shown?