Hacker News Reader

Comment by ryanlol | Hacker News Reader

This guy has had so many issues with various crawlers "attacking" his (boring!) sites, I'm starting to think he might just be a bit paranoid.

see:

https://www.hackerfactor.com/blog/index.php?/archives/678-Bo...

https://www.hackerfactor.com/blog/index.php?/archives/762-At...

https://www.hackerfactor.com/blog/index.php?/archives/775-Sc...

https://www.hackerfactor.com/blog/index.php?/archives/777-St...

https://www.hackerfactor.com/blog/index.php?/archives/779-Be...

I've done lots of portscanning of the entire internet, this always generates vast amounts of abuse emails. Many of these these emails are xenophobic rants from obviously distressed people staring at their access.logs, thinking they're being targeted by the Russian government or something.

I can't help but wonder if this might be a similar case.

View on HN

On one of my obscure but public domains I always have fun seeing Chinese traffic guessing the admin password of the VPS at least 3x a day. I guess somebody is continuously poking at vulnerabilities to get root access to as many domains as they can, then later maybe selling them to their "customers"? One doesn't have to be paranoid these days...

Anybody with a gbit line will be able to easily bruteforce the entire internet, there are lots of people doing that.

This is just normal background noise on the internet, one has to be pretty paranoid to assign any significance to it.

True, you'll get all kinds of weird traffic like that running public services. Especially hidden services on Tor where people generally aren't backed by CDNs and attacks can be made anonymously.

But it's weirder to claim that Googlebot is aggressively DDoSing you.

The imgur thing looks like Google was following links from another webpages. i.e. I submit a pic to fotoforensics, I post the link here (something like fotoforensics.com/analyze?url=imgur.com/blabla) and Google ended up crawling all those links in batches.

The fact that he says something like "a Google administrator tracked down a rogue employee and made him stop" makes it clear he's paranoid (or malicious).

> The fact that he says something like "a Google administrator tracked down a rogue employee and made him stop" makes it clear he's paranoid (or malicious).

Or sensationalism. (it did land a top spot on HN)

If you read the original post about that issue [0], the imgur.com URLs submitted were for imgur web pages containing an image, rather than direct image URLs. This caused the FotoForensics service to emit an error message. Users wouldn't have any reason to link to those error pages.

[0] https://www.hackerfactor.com/blog/index.php?/archives/484-Go...

Well, there's always the saying "It's not paranoia if they're really out to get you." And DOS attacks of many flavors are employed early and often against any public site.

And regardless of the cause, someone at Google has messed up by either letting themselves be used or screwing up the crawling algorithms to create so much load against one fairly minor blog.

He's complaining that Google is sending 100 requests per hour to his site (i.e. one request every 36 seconds).

That's hardly a DOS attack. I doubt if anyone would even notice, except that he has his server set up to alert on every 404. Sure sounds like paranoia to me.

No, he's complaining that they're generating 100 alerts per hour. And that Google crawlers consume over 25% of his total available resources.

That's potentially a lot of money wasted on nothing.

The reason it's generating 100 alerts per hour is that his server alerts on every request to a "bad" (non-existent?) URL:

> My blog logs an alert when someone accesses a bad URL. The reason for the alert: a series of bad URL requests is often an indicator of an attack, or a precursor to an attack.

And he doesn't say it's using 25% of his available resources -- he says 25% of visits to his blog are from Googlebot. Since he doesn't mention what his total visits are, we have no way of knowing how much money is wasted, but I'd be willing to bet the amount is pretty close to $0.00.

Curiously, I've seen similar posts in various hacker blogs a couple of times. It's typically a very stereotypical hacker, running his own website from a machine that he personally administers, that runs only free software. Because his (and, unsurprisingly, all of them are men) other hobbies are very specific and rare, and he's very competent in them, he getting a reasonable amount of search traffic, but never a lot of comments.

I've never seen anyone who doesn't fit this stereotype posting analysis of URL hits. It's kind of seems that only this kind of person (which I actually like a lot — these guys are very fun when you have common interests) would be so anal about this.

Reply on news.ycombinator.com