Exactly. But as you say, there are so many people riding the hype wave that it is difficult to come to a sober discussion. LLMs are a new tool that is a quantum leap but they are not a silver bullet for fully autonomous development.
It can be a joy to work with LLMs if you have to write the umpteenth javascript CRUD boilerplate. And it can be deeply frustrating once projects are more complex.
Unfortunately I think benchmaxxing and lm-arena are currently pushing into the wrong direction. But trillions of VC money are at stake and leaning back, digesting, reflecting and discussing things is not an option right now.