JS devs were kinda able to patch around the nonsense because they were able to feature-detect - part of the reason this stuck around was because no legitimate user or dev cared (or should care). But the header was mostly (useless) noise, and the people spoofing were dealing with the couple bad apples of the time.
Of course, defining features is easier said than done, and a standards body is a challenging environment to define these in...
I get why people are fingerprinting bots and others are working around it, but neither are "legitimate" applications - if your content is public, it's public, end of story. And working around these controls to sell botnet access to sites is equally illegitimate - nobody has a right to resell content they do not own...