Crawling is harder than it looks

Great document!

I architected and implemented a crawler, reading this documents really confirmed problems encountered and taught me a thing or two on how to improve.

This document goes the same scope as: Why Writing Your Own Search Engine is Hard - by Anna Patterson, Stanford University (http://www.acmqueue.org/modules.php?name=Content&pa=show...) for those interested in search engines.

That probably explains why most humans prefer to walk, once they've developed the necessary motor skills. It's a much more efficient way to move and puts a lot less strain on the wrists and knees.

(Just in case someone mistakes this for seriousness, I did RTFA)