Nginx, it's nothing special it's just my load balancer.
if ($http_user_agent ~* (list|of|case|insensitive|things|to|block)) {return 403;}
if ($http_user_agent ~* (list|of|case|insensitive|things|to|block)) {return 403;}
Hardly... the article links says that a 403 will cause Google to stop crawling and remove content... that's the desired outcome.
I'm not trying to rate limit, I'm telling them to go away.
> If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really).
It would be interesting if you had any data about this, since you seem like you would notice who behaves "better" and who tries every trick to get around blocks.
I've used this with voip scanners.
I contacted the network team at Cloudflare to apologise and also to confirm whether Facebook did actually follow the redirect... it's hard for Cloudflare to see 2PB, that kind of number is too small on a global scale when it's occurred over a few hours, but given that it was only a single PoP that would've handled it, then it would've been visible.
It was not visible, which means we can conclude that Facebook were not following redirects, or if they were, they were just queuing it for later and would only hit it once and not multiple times.