Problems caused by Google crawlers | Hacker News Reader

Problems caused by Google crawlers

hackerfactor.com

142 pointsMBCook49 comments

This guy has had so many issues with various crawlers "attacking" his (boring!) sites, I'm starting to think he might just be a bit paranoid.

see:

https://www.hackerfactor.com/blog/index.php?/archives/678-Bo...

https://www.hackerfactor.com/blog/index.php?/archives/762-At...

https://www.hackerfactor.com/blog/index.php?/archives/775-Sc...

https://www.hackerfactor.com/blog/index.php?/archives/777-St...

https://www.hackerfactor.com/blog/index.php?/archives/779-Be...

I've done lots of portscanning of the entire internet, this always generates vast amounts of abuse emails. Many of these these emails are xenophobic rants from obviously distressed people staring at their access.logs, thinking they're being targeted by the Russian government or something.

I can't help but wonder if this might be a similar case.

The imgur thing looks like Google was following links from another webpages. i.e. I submit a pic to fotoforensics, I post the link here (something like fotoforensics.com/analyze?url=imgur.com/blabla) and Google ended up crawling all those links in batches.

The fact that he says something like "a Google administrator tracked down a rogue employee and made him stop" makes it clear he's paranoid (or malicious).

> The fact that he says something like "a Google administrator tracked down a rogue employee and made him stop" makes it clear he's paranoid (or malicious).

Or sensationalism. (it did land a top spot on HN)

If you read the original post about that issue [0], the imgur.com URLs submitted were for imgur web pages containing an image, rather than direct image URLs. This caused the FotoForensics service to emit an error message. Users wouldn't have any reason to link to those error pages.

[0] https://www.hackerfactor.com/blog/index.php?/archives/484-Go...

Curiously, I've seen similar posts in various hacker blogs a couple of times. It's typically a very stereotypical hacker, running his own website from a machine that he personally administers, that runs only free software. Because his (and, unsurprisingly, all of them are men) other hobbies are very specific and rare, and he's very competent in them, he getting a reasonable amount of search traffic, but never a lot of comments.

I've never seen anyone who doesn't fit this stereotype posting analysis of URL hits. It's kind of seems that only this kind of person (which I actually like a lot — these guys are very fun when you have common interests) would be so anal about this.

On one of my obscure but public domains I always have fun seeing Chinese traffic guessing the admin password of the VPS at least 3x a day. I guess somebody is continuously poking at vulnerabilities to get root access to as many domains as they can, then later maybe selling them to their "customers"? One doesn't have to be paranoid these days...

Anybody with a gbit line will be able to easily bruteforce the entire internet, there are lots of people doing that.

This is just normal background noise on the internet, one has to be pretty paranoid to assign any significance to it.

True, you'll get all kinds of weird traffic like that running public services. Especially hidden services on Tor where people generally aren't backed by CDNs and attacks can be made anonymously.

But it's weirder to claim that Googlebot is aggressively DDoSing you.

Well, there's always the saying "It's not paranoia if they're really out to get you." And DOS attacks of many flavors are employed early and often against any public site.

And regardless of the cause, someone at Google has messed up by either letting themselves be used or screwing up the crawling algorithms to create so much load against one fairly minor blog.

He's complaining that Google is sending 100 requests per hour to his site (i.e. one request every 36 seconds).

That's hardly a DOS attack. I doubt if anyone would even notice, except that he has his server set up to alert on every 404. Sure sounds like paranoia to me.

No, he's complaining that they're generating 100 alerts per hour. And that Google crawlers consume over 25% of his total available resources.

That's potentially a lot of money wasted on nothing.

The reason it's generating 100 alerts per hour is that his server alerts on every request to a "bad" (non-existent?) URL:

> My blog logs an alert when someone accesses a bad URL. The reason for the alert: a series of bad URL requests is often an indicator of an attack, or a precursor to an attack.

And he doesn't say it's using 25% of his available resources -- he says 25% of visits to his blog are from Googlebot. Since he doesn't mention what his total visits are, we have no way of knowing how much money is wasted, but I'd be willing to bet the amount is pretty close to $0.00.

What a weird... rant. There are so many questions here:

- Is he sure it's Googlebot and not somebody spoofing headers?

- Is he sure Google is generating the queries, as opposed to Google just following URL's coming from elsewhere?

- Is he sure it's the search crawler, and not some browser service trying to cache autocomplete combinations or results?

- Has he tried simple potential fixes like changing GET to POST for searches, or robots.txt to set a crawl interval to something like 30s?

It just seems so weird. Google crawlers have finite resources so this seems like a misunderstanding or really bizarre edge case or both.

He doesn't mention robots.txt once, I checked. GoogleBot not only obeys that, it also tries to detect if a site is getting overloaded, so it doesn't run it into the ground.

It's not clear if he was getting hit by GoogleBot or e.g. FeedFetcher, which does NOT obey robots.txt, because it's triggered by human actions: https://support.google.com/webmasters/answer/178852?hl=en

There's also the possibility, as you say, that traffic was spoofed. Google publishes its netblocks over DNS, so it's easy to double check that.

> Is he sure it's Googlebot and not somebody spoofing headers?

In the author's previous post on the subject [0], he verified that the request were coming from Google IPs.

> Is he sure Google is generating the queries, as opposed to Google just following URL's coming from elsewhere?

From the URLs quoted in the article, it looks more like some sort of (machine learning-y?) system trying to infer his site's canonical URL scheme.

[0] https://www.hackerfactor.com/blog/index.php?/archives/484-Go...

Is it possible that it is Google Cloud (console) IP?.

I've ran a large scale web crawler for a decade:

http://www.datastreamer.io/

The main issues and complaints we run into are mostly unusual requests for indexing content.

People see the total number of requests and think it's a significant burden and just have an irrational need to protect their content.

It's almost a form of paranoia at some point.

I mean we have NO problem removing someone that doesn't want to be indexed and 99/100 we do so without a problem.

The only times we push back are when government organizations contact us as we feel we have a legal right to index the content. We haven't had to really fight over this because they usually agree and that's the end of the discussion.

I see hundreds of thousands of requests from search engines daily without any burden, but I do block all other bots. It has nothing to do with paranoia, a lot of bots are just evil making it impossible to act towards bots in good faith. For example there are forensic bots, copyright protection bots, brand protections bots and similar bots with the whole purpose of using crawled data to send you threats and "abuse" reports or drag you to court. There are bots that scan for vulnerabilities to exploit and bots that scan pages to find something bad to blacklist somewhere. There are bots that generate spammy links or even grab content for black hat seo, something search engines punish you for. There are bots that belong to companies that sell your data or try to gain competitive advantage through this without giving you anything in return, something you may or may not be comfortable with. There are even bots that gather data for political and intelligence purposes, literally causing death and destruction.

Given the wide range of bots you mentioned, I’m very curious how you’re just blanket blocking all other bots? Could you elaborate a bit?

Edit: add other; ambiguous.

Here's the robots.txt file for hackerfactor.com:

  User-agent: ia_archiver
  Allow: /
  User-agent: ScoutJet
  Disallow: /blog
  User-Agent: Googlebot
  Allow: /
  Disallow: /blog/index.php?/archives/2018/
  Disallow: /blog/index.php?/archives/2017/
  Disallow: /blog/index.php?/archives/2016/
  Disallow: /blog/index.php?/archives/2015/
  Disallow: /blog/index.php?/archives/2014/
  Disallow: /blog/index.php?/archives/2013/
  Disallow: /blog/index.php?/archives/2012/
  Disallow: /blog/index.php?/archives/2011/
  Disallow: /blog/index.php?/archives/2010/
  Disallow: /blog/index.php?/archives/2009/
  Disallow: /blog/index.php?/archives/2008/
  Disallow: /blog/index.php?/archives/2007/
  Disallow: /blog/index.php?/archives/2006/
  Disallow: /blog/index.php?/archives/2005/
  Disallow: /blog/index.php?/archives/2004/
  Disallow: /blog/index.php?/archives/2003/
  Disallow: /blog/index.php?/archives/2002/
  Disallow: /blog/index.php?/archives/2001/
  Disallow: /blog/index.php?/archives/2000/
  #Disallow: /blog/index.php?/categories/
  User-agent: *
  Disallow: /blog
  Disallow: /badbot-a2d0ac98abcaf3cafd9eff83b3cffa98fec7a390a6c5b9

I'm not 100% certain about that ? in there, but I think you can do this instead:

  Disallow: /blog/index.php?/archives/*/

The first wave of the attack began dropping off letters.

I wouldn't call it an attack, but it seems more like some sort of autocompletion checking given how similar it is to the queries generated to Google if you use their search autocomplete feature --- one request for each character entered.

That said, Google's search quality has already taken a steep decline and I've been getting blocked very often for searching more obscure things now, so whatever "bot mitigations" they put in, I hope they don't make that even worse.

> I've been getting blocked very often for searching more obscure things

Me too! It seems like nearly any time I go past the second page for results they think I'm a bot.

The claim was that it started with the full URL and progressively removed them. If it were reversed I would buy the autocomplete bug. I fixed something like that a few weeks ago at work.

If Chrome address bar autocomplete generates queries (which I'm not sure if it does), then the following scenario could easily explain what's going on:

- As a user, I type "foo.com" into my address bar. Having already gone to a specific page of that site before, autocomplete "helpfully" expands it to that page's URL "https://foo.com/this/is/a/page"

- It turns out I don't actually want that, I want the main page, so I start deleting characters until I'm left with "https://foo.com"

If Google took each of those deletions as a separate URL (either by bug or on purpose) and tried to verify said URL is valid, I could see a GoogleBot request being generated for each iteration between the initial URL and the full URL, starting with that initial one.

Given what I've read on the article, I'm fairly confident that what's happening here is a conjunction of Chrome autocomplete suggestions causing the crawl of every URL permutation and the website returning the same content to every permutation, instead of using 403 redirects or including a `rel=canonical tag`.

Some of these mitigation tactics seem a little much, given the situation; did the author consider a robots.txt first?

"making my web site less usable is the only way I can find to stop the Google from indexing infinite trees of it's own creation"

So apparently not.

Are you claiming that Google doesn't respect robots.txt? He can just disallow *.

Presumably you'd want Google to index pages, just not try to discover pages by brute forcing the URL?

Is there any way to do that in robots.txt? Someone did mention adjusting the crawl rate in Webmaster Tools -- would the author need to do this for every search engine?

He can submit a sitemap to Google if he’d like a particular structure crawled.

Sounds like his issue is mainly with Google, not other search engines, so it’s not an O(n) problem he’s got to solve.

Submitting a sitemap might cause Google to crawl those pages, but it certainly won't stop google from crawling additional pages like the ones mentioned in the blog post. Lots of sitemaps are incomplete or outdated or completely invalid or whatever.

I feel like you're still misunderstanding just how simple the fix is. If he wants to, he can just disallow the portions of his site that he claims Google is scraping via a robots.txt, and he can also submit a sitemap to the portions that he does want crawled.

OK, so let's look at the top example of unwanted crawling from the article:

  /blog/index.php?/categories/14-Forensics (wanted)
  /blog/index.php?/categories/14-Forensic (unwanted)
  /blog/index.php?/categories/14-Forensi (unwanted)
  /blog/index.php?/categories/14-Forens (unwanted)

How do you propose blocking the last 3, without having an enormous robots.txt?

I think if you do a Disallow: * and also submit an explicit sitemap, then the sitemap would 'win' for those explicit URLs, no?

You can return a `X-Robots-Tag: none` header for the inner URLs. It's a thing I throw into APIs I write, just in case Google manages to find it.

robots.txt can certainly keep google away from your search box, but it won't defend you against the lower-case probes, the extra letter probes, the delete-a-character probes, etc.

Yet he didn't try it. That's seriously weird that he doesn't mentions trying that (he updated his post to say they doesn't support regex, yet does support wildcard, which is all what's needed for a search).

He removed the search before using a simple robots.txt...

I had no idea that googlebot (and others) will sometimes POST to urls[0] - this seems like a terrible idea.

I imagine what happened to this guy is that somebody coded up a site that used his as a service and google spidered it and went crazy trying to follow the links. It sucks, but that is what robots.txt is for. Anything that requires more than just serving a page should be blocked.

You can drive yourself insane reading your logs trying to guess what bots are trying to achieve. They seem to excel at spidering exactly what you don't want while ignoring the content that would actually be useful.

The one time I did have a problem with a misbehaving crawler was with something called 80legs[1]

[0] https://webmasters.googleblog.com/2011/11/get-post-and-safel...

[1] https://sheep.horse/2012/8/80legs_is_a_pain_in_the_neck.html

That is pretty crazy, I'm kind of shocked.

I'm sure their ML is pretty decent... but I do have to wonder if it's ever wound up leaving hundreds of comments on an anonymous forum or something, thinking it was a search box.

This sounds to me like someone is simply using GoogleBot in their request headers whilst probing.

You can adjust the crawl rate in webmaster tools

I'm surprised by almost all comments in this thread dismissing the article completely.

Google does field trials all the way in Chrome. Why would it be so unrealistic that they did similar trials with Googlebot?

Google bot is stupid af and their support is even less helpful.

TL;DR: Google just scans you site even if it always gets 404s for millions of requests.

> Unfortunately, I had to remove that feature because Google began submitting random dictionary words. Hundreds of thousands of them.

That sounds insane. Is that actually true or is there an exaplanation of some sort?

Googlebot does sometimes submit search forms in an attempt to discover new pages, though I'm surprised to hear "hundreds of thousands". Either that's an exaggeration or it was over a very long time period.

As other comments mentioned, he could have easily prevented this with robots.txt:

    User-agent: *
    Disallow: /search

Would that work if the search box was on the main page, though? Presumably, GoogleBot wouldn't index the results, but is it smart enough to know that the form redirects to the disallowed page?

Yes, Googlebot (and everyone else, really) is smart enough to figure out if a form action is disallowed in robots.txt.

A good start:

https://support.google.com/webmasters/answer/80553?hl=en

Then use robots.txt on any dynamic route like search.

A solid site map could help too.

What's surprising to me is how many hoops people have to jump through just to tell Google to go away.

Reply on news.ycombinator.com