Hacker News Reader

Comment by jmpe | Hacker News Reader

How would one differentiate between a legitimate query and one that traps? Seriously, suppose I link to a search result page for MySQL [on some site] for info on such a query, wouldn't it look the same?

Not looking for work atm, so my advice: this is not your problem.

View on HN

A typical crawler trap is a link to a page which adds some sort of "info" the url, so lets say you have a URL like

    http://not-a-real-host.com/some_page/

And that page has a link for comments which takes you to

   http://not-a-real-host.com/some_page/?show_comments=1

And the designer didn't neuter the show comments link and if you follow it, it takes you to:

   http://not-a-real-host.com/some_page/?show_comments=1&show_comments=1

Ad infiniteum. That is a bug but the same sort loop where the cgi arg is going from ?page=1, ?page=2 is valid.

Basically folks who love working on web crawlers are people who like figuring out puzzles like this. The number of interesting things web sites can do that make crawling them not work is unlimited I believe. We have lots of regular checks (things like how many unique documents have I pulled from a host, or how many pages have the same MD5 hash, etc.) Designing defenses is part of the fun.

He is asking how you plan to differentiate when it is ok for the query to be in the url (perhaps it is checked and validated with a regex on the server so it cant be abused) or when its actually capable of doing something malicious the maintainer is not aware of.

Reply on news.ycombinator.com