Comment by feelin_googley | Hacker News Reader

Comment by feelin_googley | Hacker News Reader

Here is an example of the "good bot"/"bad bot" nonsense in action.

This is an article about the LinkedIn v hiQ case at AdWeek.

  curl --user-agent INSERT_ANYTHING_HERE http://www.adweek.com/digital/rami-essaid-distil-networks-guest-post-linkedin-hiq-labs/

It seems AdWeek can distinguish a "good bot" from a "bad bot" irrespective of the behavior of the user^W bot, i.e., whether it is one single HTTP request or 10,000 consecutive requests is irrelevant.

How do they do it?

Pattern match against the User-Agent string.

Effective shibboleth.^W engineering.

Clarification: If a user, not a "bot", makes the "wrong" choice of user-agent string (e.g. in the browser settings), then they will be labeled a "bad bot", even if their behavior is no different than other users who are not labeled "bad bots". For example, they make one HTTP GET request just like any other user. There are databases of "acceptable" user-agent strings available to anyone. If still unsure about the point I am making, see this post from several days ago: https://www.sigbus.info/software-compatibility-and-our-own-u...

What would be a better solution, IP address check to allow only known google crawlers perhaps?

The real solution is disallowing behaviors, instead of shibboleths.

It's surprising that malicious bots aren't exploiting those things already.

Classify IPs based on their recent behavior[2]. Most bots behave very differently from the median user, along many different dimensions -- volume of requests, time between requests, visit length, which links are followed, etc.

And if this means that bots are altered to become indistinguishable from users, and therefore have a minimal impact on a site's loading? Well, mission accomplished[1].

[1] https://xkcd.com/810/

ETA: [2] Recent behavior (as opposed to all historical behavior) is used so that someone inheriting a "bad" IP isn't completely screwed over.

That's a superb xkcd that I hadn't seen yet, thanks.

That practically invites them to present a different page to google as to a normal user, the former pure SEO, the latter perhaps pure advertising.

And Google will happily deindex the site as soon as they find out

Raising an interesting question: can a website owner (use the law to) ban google from accessing their website by any mechanism other than their crawler in order that google doesn’t find out?

Sure, obviously limited utility just like “the right to be forgotten”s flaw of diffing USA internet from EU internet to find specifically what people want forgotten, but shenanigans interest me.

A related story is that windows 9 isn't a thing because software used to check for windows 95 and 98 by matching the name to "windows 9".

Reply on news.ycombinator.com