Within 15 minutes of setting up a https CI environment, complete with robots.txt, Googlebot was hitting the DNS name which also wasn't public or previously used or easily guessed.
All the team use Gmail and one was using Chrome.
All the team use Gmail and one was using Chrome.
I remember a storm in a teacup about MS doing this via their browser toolbar.
Recently, they started tracking all downloads made on Chrome (for malwares), it includes the filename, the URL, IP and the timestamp. Sucks hard since I love Chrome and the only way to disable it is to disable the website malware checker (which only uses part of the hashes anyway).
For a lot of people spending >$100/year and many hours maintaining it isn't worth it.