Well, there’s a bigger issue of AIs not doing what we tell them to do, but I think that’s orthogonal to this UI issue. If you want to fault the model for not obeying the prompt or for a poor solution, then fine. My point is, it solved the problem. If the API isn’t there, then that eliminates the API as a solution, but it says nothing about how well the model can solve the UI problem. Perhaps the model can’t solve it. If we really want to benchmark the ability of models to use human UIs to solve problems, then perhaps we need to choose benchmarks that don’t have APIs available such that the model cannot get “creative” in any way and must use the UI as part of the task. More simply, maybe the model isn’t the problem; maybe the benchmark designer is.