Also, keep in mind that google has recently changed the way they deal with robots.txt. So they only way to do what the author is suggesting is to add the noindex meta tag and let them crawl. Once they are removed from the index, block the google IPs.
Of course they still have it indexed in their own systems .... would you trust Google not to.
To block those, you can feed their IP's to haproxy or iptables ipset.
fetch_google()
{
for line in $(dig +short txt _cloud-netblocks.googleusercontent.com | tr " " "\n" | grep include | cut -f 2 -d :)
do
dig +short txt "${line}"
done | tr " " "\n" | grep ip4 | cut -f 2 -d : | sort -n | uniq | xz -9ecv > _GOOGLE.netset.xz
};
acl BOTS src -f bot.txt
Google is going to crawl, you just can’t stop them from crawling even if you tell them not to in robots.txt. Best option at this point is to add a noindex tag on pages and let them crawl to see it.
An implied TOS notice on a separate page or footer is not considered enforceable last time I checked, likewise for robots.txt.