Hacker News Reader

Comment by ChuckMcM | Hacker News Reader

This is sort of sad, now I have to go dig through crawler code to see what we (Blekko) do when we crawl a site. We do avoid known 'traps' (which are often inadvertent) that result in loops but this is a bit more nefarious.

On a related note, if you would like to be the person looking through the crawler code and designing a defense for this, and are willing to work in the bay area. Send me an email :-)

View on HN

I don't believe this is a problem a crawler should be solving. The site left open SQL injection vulnerabilities, and they're complaining that someone is attacking them. Whether the attack originates from a crawler or not is beside the point.

I agree that crawlers can't solve the problem, the trick is trying to avoid contributing to the problem. Take another well known example, DNS reflection attacks. Clearly DNS can't solve the DDOS "problem" but it can avoid helping out by not recursively resolving queries for untrusted clients.

So any time the crawler can be a bit more aware about what it is heading into, the saves it time not looking at pages which we know apriori we would not be indexing anyway. And that (which pages should be indexed) can only be answered from the set of pages that are crawled.

The article is about an attacker using Google's behaviour as a form of indirection. The attacker can't be traced directly by the vulnerable site if he carried out the attack simply by triggering Google's crawler.

I agree that it mentions indirection, but I don't agree that it is the purpose of the article at all. The article finishes with this line:

> We are contacting Google about it, but it is always something important to keep in the back of your mind. You can’t just whitelist their IP, and allow through without any type of inspection.

I wouldn't be surprised if the authors knew of the sqli vector, and whitelists appropriate clients. The problem is the sqli, not WHO can hit it (or who was ultimately responsible).

Edit:

I just went and looked at the company behind this article. I was too quick to judge apparently. Their business is protecting developers from stupid mistakes like sqli at the firewall.

Therefore, the article really IS about indirection. Sorry.

How would one differentiate between a legitimate query and one that traps? Seriously, suppose I link to a search result page for MySQL [on some site] for info on such a query, wouldn't it look the same?

Not looking for work atm, so my advice: this is not your problem.

A typical crawler trap is a link to a page which adds some sort of "info" the url, so lets say you have a URL like

    http://not-a-real-host.com/some_page/

And that page has a link for comments which takes you to

   http://not-a-real-host.com/some_page/?show_comments=1

And the designer didn't neuter the show comments link and if you follow it, it takes you to:

   http://not-a-real-host.com/some_page/?show_comments=1&show_comments=1

Ad infiniteum. That is a bug but the same sort loop where the cgi arg is going from ?page=1, ?page=2 is valid.

Basically folks who love working on web crawlers are people who like figuring out puzzles like this. The number of interesting things web sites can do that make crawling them not work is unlimited I believe. We have lots of regular checks (things like how many unique documents have I pulled from a host, or how many pages have the same MD5 hash, etc.) Designing defenses is part of the fun.

He is asking how you plan to differentiate when it is ok for the query to be in the url (perhaps it is checked and validated with a regex on the server so it cant be abused) or when its actually capable of doing something malicious the maintainer is not aware of.

Sounds fun, but I'm not moving to the bay area. :P

Reply on news.ycombinator.com