It obviously seemed like a good idea in 1991 but in retrospect the User-Agent header was probably a bad idea. TBL never anticipated it would be abused by servers for this, and for browser fingerprinting...