Hacker News Reader

Comment by amelius | Hacker News Reader

Nice to see fingerprinting used against Google (instead of by Google). But this is easy to fix, so I don't expect this to work for much longer.

View on HN

I think how easy it is for Google to fix to make detecting googlebot this way harder depends on why Googlebot is doing it in the first place, which we don't really know. If it's done for performance reasons, or for predictability reasons (rendering the same page twice guaranteed or at least more likely to produce the same result), it might be difficult to change without cost.

But I believe Googlebot always faithfully sends it's user-agent. Is there a reason Google would care about 'fixing' this to make Googlebot harder to detect via random() predictability, when you can always just detect it via user-agent anyway? I'm not sure, curious if others have thoughts!

Google has checks in place to see if someone is serving things to GoogleBot differently that the rest of the users. So it almost definitely has bots that double checks pages without the user-agent.

If the "disguised" googlebot is the same as the actual one, chances are it is since it would want to be as close as possible to not flag false positives, and use the same seed for consistency then you might be able to use that to avoid detection on the fact that you are serving google something different than normal users.

Newspaper used/do that to be able to have their full article content indexed while serving a paywall to everyone else.

It is much easier to detect Googlebot by IP address or User-Agent though.

I’d assume that they occasionally send requests with different IPs and user-agents to detect cloaking.

Yes, but their IPs are registered to Google Inc.

Are they?

Yes. Are you implying they are not?

Yes? I mean, scanning for cloaking only from their own IP range sounds ridiculous.

I think that cloaking is very difficult to detect automatically without using humans. You have to distinguish for example between a paywall and real cloaking.

Maybe they change the behavior of their random() function as well. The deterministic random() function can very well be a red herring.

I'd guess if they're crawling purely for the purpose of detecting cloaking (which would be a much smaller-scale job than the standard Googlebot indexing), they'd just use Chrome Headless[0][1]

[0] https://intoli.com/blog/not-possible-to-block-chrome-headles...

[1] http://antoinevastel.github.io/bot%20detection/2018/01/17/de...

I doubt that they use chrome at all. They probably just fetch the page over http and let it run in a JS sandbox. hence the deterministic random function (which I assume is not there in chrome?)

2018/01/18: Chrome 41: The Key to Successful Website Rendering: "Google updated their Search Guides and announced that they use Chrome 41 for rendering."

https://www.elephate.com/blog/chrome-41-key-to-website-rende...

It is interesting that when I use Chromium 46 (which is newer than Googlebot) I get warnings from Github and Google Docs about an outdated or unsupported browser, and I get plain HTML Github pages without JS. But Google uses even older browser than I. So even Google cannot cope with browser version race and keep their browsers updated although they require this from the users.

According to that article, Googlebot is "based on Chromium 41", it doesn't actually use Chromium 41 directly, it's still a distinct, separately developed browser.

The warning you're seeing is

a) probably shown based on UA, and Googlebot has a very different UA to the Chromium browser

b) warning users who will need to browse/actively use the site. Googlebot simply parses content, so has no use for quite a lot of the active functionality on the site. As such, it will generally not need to support all of the features used by that active functionality, just barely enough to be able to read content.

I'm sure they don't use Chrome for Googlebot, that would be quite inefficient. My comment was about cloaking detection.

Maybe that's just what they want you to think! /tinfoil

I don't think it's particularly easy to fix without fixing a lot of other things. Let's see how much time it takes them. :)

Perhaps for performance or testing reasons they compiled V8 with a predefined seed, keywords are "V8", "mksnapshot", "--random-seed". It should not be too difficult to fix it if there are no artificial restrictions by use cases.

Reply on news.ycombinator.com