Most 404 accesses are from people probing a server for common URL's belonging to vulnerable web applications.
Thus my 404 page is a honeypot, and too many 404 accesses in a short period results in an automatic IP ban.
Thus my 404 page is a honeypot, and too many 404 accesses in a short period results in an automatic IP ban.
If someone is trying to get to admin.php, sure, ban them. Or if they are not following robots.txt. But sitemaps are not reliable enough sometimes and not all crawlers are meanies.
Not my fun and profit, though.
The idea of saying yes to everything is comically explored by Jim Carey in the film:
User-agent: *
Disallow: /
User-agent: Googlebot
Allow: /
User-agent: Slurp
Allow: /
User-Agent: bingbot
Allow: /Anyone can make a crawler and then have it report as Googlebot. That doesn't even violate the robots.txt; it says, if your name is Googlebot, you're allowed.
Blocking crap requires cunning: code that looks for suspicious access patterns and responds.
A genuine Googlebot should be operating from a Google domain. If we reverse the client IP of a Googlebot request, we get something in the ".googlebot.com" domain.
Include a directory "visiting-this-will-ban-you" in your robots.txt that IP bans whomever visits it.
Heck when MSFT redesigned technet/MSDN half of the links were dead allot of the Google results for legacy products will still lead you to a 404 page on technet...