Hacker News Reader

Comment by throwasehasdwi | Hacker News Reader

Blocking access based on arbitrary user agent strings is a really bad idea. Every single bad bot will avoid known user agent strings or pretend to be Google, so you're only blocking well behaved ones. Plus there's thousands of browser versions out there, so there's a very good chance you're blocking some users for no reason.

The proper way to do this is to block by IP, based on behavior. Block IPs slowing down the site or throw up a captcha like cloudflare does.

Blocking bots sounds great but it just brings Google one step closer to a monopoly. Even good bots just pretend to be people nowadays because lots of people are implementing naive site protection strategies.

Edited: to be less mean

View on HN

Yes, you're right. There are many ways to block robots: IP, UA, behaviour analysis. An advertising company has to have UA based filtering to be compliant with standards. However, the focus of the blog post is on performance rather on how to block bots.

The article is not about filtering bad bots. It even says so right near the top:

>We won’t cover black bots because it is a huge topic with sophisticated analysis and Machine learning algorithms. We will focus on the white and grey bots that identify themselves as such.

This is about not wasting time & effort showing advertising banners to good bots.

Plus it's baking planned obsolescence into your code. As soon as you stop updating it, it will start blocking newer browser editions and versions.

> single dumbest idea

I agree with your sentiment, but you should try to be a little more constructive.

In theory you are right, but in reality 99% of the rogue bots are actually just some scraper tools were ignorant users changed sane defaults to "make it go faster". They usually don't have enough knowledge to even understand that they are routed into a black hole - not to speak of being able to do something about it.

Disclaimer: Getting rid of those idiots^wmisguides poor souls is part of my job description.

Then why not block based on bots that ignore robots text or make requests too quickly like I suggested? Looking for sub-strings in the UA header is a hack at best.

They all require one to keep state information on the server side and with overall little benefit. Yes, I do what you proposed, but in hindsight it was a waste of time to set that up: 99% simply don't know what they are doing and therefore easy to catch.

I don't care about this one person who knows to change the user agent sent, those are the ones you can usually talk to and they'll happily throttle their crawlers.

The other 99 are the problem - and they are a problem that can be solved pretty well by simple string matching.

> Every single bad bot will avoid known user agent strings

If I was writing a bot, I would set user agent to some well known and very popular value, i.e. newest Chrome on Windows, or something like that.

Reply on news.ycombinator.com