> some of the bots won't respect robots.txt

The link you posted literally only lists the scanner related to security/safe browsing. Why would it? What kind of bot would that be if an attacker could just put up a robots.txt that causes it to ignore the site?! And many do...for obvious reasons.

It's misleading because this implies you're getting heavy scraper traffic from Google bots that don't respect robots.txt for reasons of greed rather than because it's necessary so they can proactively avoid surfacing malicious websites.