Looks like Google has started rolling out this Web Bot Auth thing which seems like something that should gain adoption or become an open standard. https://developers.google.com/crawling/docs/crawlers-fetcher...

Seems like the crawler companies would be incentivized to not want to take responsibility for people spoofing their user agents.