> Really there's nothing wrong with having a UA header but it gets abused because HTTP never had any standardized way to communicate feature sets. It should have had both. A way to identify specific apps for bug workarounds etc, and a way to communicate what versions and parts of the specs are implemented.
The problems with this is that it's an adversarial market. It's not quite as bad as it used to be, now that there's almost no cases where the same company controls both the server and the browser (with the very big exception of visiting google.com). But there's still incentives to not treat different browsers the same way, or for the browser to lie about capabilities.
One silver lining of moving so much CPU-intensive site building work to clients is that feature/bug detection can be done there, live, where it has more chance of actually testing the thing you want to check for.