Most Googlebots Are Fake | Hacker News Reader

Most Googlebots Are Fake

digitalseams.com

9 pointsbobbiechen3 comments

> Chris Siebenmann theorizes that “it’s a large scale campaign by a single abusive crawler by people who can afford to obtain a lot of servers at a lot of different hosting providers”, rather than a widespread rise in popularity for an old trick.

And apparently also thinks every person using an older browser version is a bot, I guess.

https://utcc.utoronto.ca/~cks/cspace-bad/cspace-old-browser....

I don't understand why the page even links to other stuff... I can't access those either because they just redirect back to that page

That's a shame, I wonder where he draws the line.

While herding to the majority of latest Chrome makes his life easier, there really is a very long tail of real users who are not there. I get way too many challenges on my personal laptop running Firefox on Ubuntu (including infinite redirect loops).

A lot of too clever people, various web site developers have decided that those who run scrapers are really dumb and use old tools that have or can not be modified and only use old agents.

There are of course exceptions like angling sites [1] which use search engines to advertise but offer little to the web surfer without paying for access - the only way for the average user to then see what google cached and validate if the site is worthwhile, is to then lie and pretend to be a bot. However I have found many of these types of sites have since brought in additional tactics to thwart this practice.

I get the same referral "old browser" ... It's something I do see a lot as I use an older browser with extensions, simply for the fact it displays more information when browsing that I plainly don't see in various latest and greatest browsers - thus at times I do have to shift browsers, and even then at times modify my browser agent [2] [3] if I need to visit a particular site with that browser - only under duress do I go down this path, most of the times I will simply dismiss such sites ... or open up a portable recently updated browser eg Supermium which by the way opens that link fine - of course nothing important there, just casual observational blah blah comments.

I don't know what's going to happen when scraping tools are able to totally fake being a Windows 11 system with the latest greatest browser with all the faked PII fields, baked cookies displayed perfectly /s

[1] use of lure, as in fishing.

[2] https://www.howtogeek.com/113439/how-to-change-your-browsers...

[3] (about the same as [2] but perhaps with more modern interfaces) https://geekflare.com/dev/change-user-agent-in-browser/

Reply on news.ycombinator.com