"I often am repulsed that servers reads user agent." That is a backwards way of looking at it, they dont take the user agent, you give it. You you give any user agent you like or none.

No one providing a free resource online has any obligation to treat your requests in a particular way. If you dont like the way a particular HTTP service is responding, then dont send it requests.